[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:15:47 dist-test-slave-gjrz krb5kdc[30630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:15:47 dist-test-slave-gjrz krb5kdc[30630](info): set up 2 sockets
Aug 10 15:15:47 dist-test-slave-gjrz krb5kdc[30630](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 15:15:48.346172 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:15:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:15:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:15:48.466858 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42867
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:48.552467 30649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:48.552676 30649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:48.552698 30649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:48.553797 30649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:15:48.553830 30649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:48.553843 30649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:48.553854 30649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:15:48.553864 30649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:15:48.555472 30649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:48.555842 30649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:48.556075 30649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:48.562537 30659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:48.562537 30654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:48.562606 30656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:48.563766 30649 server_base.cc:1034] running on GCE node
I20260810 15:15:48.564170 30649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:48.564590 30649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:48.565730 30649 hybrid_clock.cc:648] HybridClock initialized: now 1786374948565729 us; error 53 us; skew 500 ppm
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:48.568578 30649 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:15:48.569257 30649 webserver.cc:460] Webserver started at http://127.29.230.62:37187/ using document root <none> and password file <none>
I20260810 15:15:48.569494 30649 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:48.569541 30649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:48.569686 30649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:48.571069 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "ab643e0281484ef3a3c9e4dba7e3972e"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "de2add2c056a90c0c66629ed78d67d71"
server_key_iv: "cb9f4c99f85116aba8aaa25d6610c3f7"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.571470 30649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "ab643e0281484ef3a3c9e4dba7e3972e"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "de2add2c056a90c0c66629ed78d67d71"
server_key_iv: "cb9f4c99f85116aba8aaa25d6610c3f7"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.573455 30649 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 15:15:48.574496 30671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.574747 30649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:48.574846 30649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "ab643e0281484ef3a3c9e4dba7e3972e"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "de2add2c056a90c0c66629ed78d67d71"
server_key_iv: "cb9f4c99f85116aba8aaa25d6610c3f7"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.574911 30649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:48.602195 30649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:48.605054 30649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:48.605301 30649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:48.611021 30649 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:42867
I20260810 15:15:48.611145 30775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:42867 every 8 connection(s)
I20260810 15:15:48.611811 30649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:15:48.612869 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30649
I20260810 15:15:48.612986 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:15:48.613472 30777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:48.613852 30616 external_mini_cluster.cc:1428] Setting key f400f7062f40baeaec4c03c752fc575b
I20260810 15:15:48.617763 30777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: Bootstrap starting.
I20260810 15:15:48.618922 30777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:48.619493 30777 log.cc:824] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:48.620448 30777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: No bootstrap required, opened a new log
I20260810 15:15:48.623136 30777 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } }
I20260810 15:15:48.623375 30777 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:48.623490 30777 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab643e0281484ef3a3c9e4dba7e3972e, State: Initialized, Role: FOLLOWER
I20260810 15:15:48.623675 30777 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } }
I20260810 15:15:48.623831 30777 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:15:48.623862 30777 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:15:48.623914 30777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:48.624570 30777 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } }
I20260810 15:15:48.624804 30777 leader_election.cc:302] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab643e0281484ef3a3c9e4dba7e3972e; no voters:
I20260810 15:15:48.625063 30777 leader_election.cc:288] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:15:48.625149 30785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:48.625360 30785 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [term 1 LEADER]: Becoming Leader. State: Replica: ab643e0281484ef3a3c9e4dba7e3972e, State: Running, Role: LEADER
I20260810 15:15:48.625562 30785 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } }
I20260810 15:15:48.625572 30777 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:15:48.627621 30787 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } } }
I20260810 15:15:48.627775 30787 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [sys.catalog]: This master's current role is: LEADER
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:48.628038 30788 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab643e0281484ef3a3c9e4dba7e3972e. Latest consensus state: current_term: 1 leader_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab643e0281484ef3a3c9e4dba7e3972e" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42867 } } }
I20260810 15:15:48.628110 30788 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e [sys.catalog]: This master's current role is: LEADER
W20260810 15:15:48.628247 30809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:15:48.628295 30809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:15:48.628417 30814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:15:48.628742 30814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:15:48.631116 30814 catalog_manager.cc:1347] Generated new cluster ID: ffe8fdad96634e7691d5ff265c153ce1
I20260810 15:15:48.631201 30814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:15:48.632990 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:48.615364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.29.230.62:42867)
0810 15:15:48.615689 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:48.615698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:48.615901 (+ 203us) server_negotiation.cc:407] Connection header received
0810 15:15:48.618703 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:48.618709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:48.618817 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:48.618947 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:48.621153 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.622115 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.623662 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.624152 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.624245 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:48.628830 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:48.628835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:48.628835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:48.628847 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:48.630835 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.631331 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.631334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.631334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.631448 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.631772 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.631774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.631774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.632108 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:48.632245 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:48.632560 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 15:15:48.632751 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260810 15:15:48.643846 30814 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:15:48.644400 30814 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:15:48.649268 30814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab643e0281484ef3a3c9e4dba7e3972e: Generated new TSK 0
I20260810 15:15:48.649441 30814 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:15:48.678190 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:48.783387 30821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:48.783596 30821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:48.783622 30821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:48.784869 30821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:48.784910 30821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:48.784943 30821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:15:48.786427 30821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:48.786728 30821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:48.786964 30821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:48.792141 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:48.792281 30821 server_base.cc:1034] running on GCE node
W20260810 15:15:48.792239 30838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:48.792239 30840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:48.792649 30821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:48.792914 30821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:48.794056 30821 hybrid_clock.cc:648] HybridClock initialized: now 1786374948794040 us; error 35 us; skew 500 ppm
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:48.796818 30821 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:15:48.797286 30821 webserver.cc:460] Webserver started at http://127.29.230.1:36837/ using document root <none> and password file <none>
I20260810 15:15:48.797483 30821 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:48.797530 30821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:48.797654 30821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:48.798688 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "e9f0ea3cc9f646678baa7dd0ace50ebe"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "d1eec13d14923e58ab781a707054e08f"
server_key_iv: "f8bba9bef20b24bd2b0734bae87178e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.799018 30821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "e9f0ea3cc9f646678baa7dd0ace50ebe"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "d1eec13d14923e58ab781a707054e08f"
server_key_iv: "f8bba9bef20b24bd2b0734bae87178e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.800410 30821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:15:48.801209 30849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.801398 30821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:48.801514 30821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "e9f0ea3cc9f646678baa7dd0ace50ebe"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "d1eec13d14923e58ab781a707054e08f"
server_key_iv: "f8bba9bef20b24bd2b0734bae87178e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.801584 30821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:48.812511 30821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:48.814406 30821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:48.814553 30821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:48.814841 30821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:48.815237 30821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:48.815375 30821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.815429 30821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:48.815454 30821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.821338 30821 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:35231
I20260810 15:15:48.821383 31043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:35231 every 8 connection(s)
I20260810 15:15:48.821702 30821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:15:48.823664 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30821
I20260810 15:15:48.823750 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:15:48.823880 30616 external_mini_cluster.cc:1428] Setting key fbc4eb173eb814728152305a5a7ecaa5
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:48.832592 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:48.822640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:46449 (local address 127.29.230.62:42867)
0810 15:15:48.822698 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:48.822700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:48.822881 (+ 181us) server_negotiation.cc:407] Connection header received
0810 15:15:48.823440 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:48.823442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:48.823455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:48.823517 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:48.824811 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.825350 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.826701 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.826984 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.827047 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:48.829668 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:48.829670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:48.829670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:48.829676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:48.831098 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.831584 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.831585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.831586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.831682 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.831933 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.831934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.831935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.832051 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:48.832107 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:48.832406 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 15:15:48.832500 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:15:48.832934 31048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:48.822595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42867 (local address 127.29.230.1:46449)
0810 15:15:48.822758 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:48.822796 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:48.823336 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:48.823593 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:48.823597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:48.824438 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:48.824716 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:48.824723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.825445 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.825446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:48.826603 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:48.826607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.827081 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.827082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:48.827283 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:48.827838 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:48.827848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:48.829578 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:48.831169 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:48.831195 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:48.831198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:48.831507 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:48.831756 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:48.831757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:48.831757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:48.831865 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:48.832137 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:48.832144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:48.832275 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 15:15:48.832361 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 15:15:48.832521 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:15:48.833515 31045 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42867
I20260810 15:15:48.833604 31045 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:48.833776 31045 heartbeater.cc:511] Master 127.29.230.62:42867 requested a full tablet report, sending...
I20260810 15:15:48.834739 30708 ts_manager.cc:194] Registered new tserver with Master: e9f0ea3cc9f646678baa7dd0ace50ebe (127.29.230.1:35231)
I20260810 15:15:48.835572 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:46449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:15:48.852067 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:48.934093 31055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:48.934301 31055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:48.934332 31055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:48.935485 31055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:48.935524 31055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:48.935559 31055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:15:48.937016 31055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:48.937244 31055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:48.937449 31055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:48.942408 31070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:48.942507 31073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:48.942607 31055 server_base.cc:1034] running on GCE node
W20260810 15:15:48.942440 31065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:48.942838 31055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:48.943048 31055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:48.944205 31055 hybrid_clock.cc:648] HybridClock initialized: now 1786374948944191 us; error 28 us; skew 500 ppm
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:48.946743 31055 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:15:48.947198 31055 webserver.cc:460] Webserver started at http://127.29.230.2:44769/ using document root <none> and password file <none>
I20260810 15:15:48.947419 31055 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:48.947466 31055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:48.947590 31055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:48.948664 31055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "59376d7f4a014ea19d02cf0b0d233398"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "7455b844c902fc0eca9b80499cedfd69"
server_key_iv: "59bc8c5a56037dc0163113ab4ab5d47a"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.949003 31055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "59376d7f4a014ea19d02cf0b0d233398"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "7455b844c902fc0eca9b80499cedfd69"
server_key_iv: "59bc8c5a56037dc0163113ab4ab5d47a"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.950340 31055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:15:48.950994 31084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.951140 31055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:48.951225 31055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "59376d7f4a014ea19d02cf0b0d233398"
format_stamp: "Formatted at 2026-08-10 15:15:48 on dist-test-slave-gjrz"
server_key: "7455b844c902fc0eca9b80499cedfd69"
server_key_iv: "59bc8c5a56037dc0163113ab4ab5d47a"
server_key_version: "encryptionkey@0"
I20260810 15:15:48.951313 31055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:48.961591 31055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:48.963658 31055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:48.963790 31055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:48.964038 31055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:48.964326 31055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:48.964359 31055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.964386 31055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:48.964411 31055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:48.970156 31055 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:33563
I20260810 15:15:48.970199 31268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:33563 every 8 connection(s)
I20260810 15:15:48.970525 31055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:15:48 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374948, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:48.977442 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31055
I20260810 15:15:48.977552 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:15:48.977738 30616 external_mini_cluster.cc:1428] Setting key 5e7f926ee328d624e0b1aa63b6c7d743
I20260810 15:15:48.979789 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:48.971311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:48695 (local address 127.29.230.62:42867)
0810 15:15:48.971373 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:48.971375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:48.971586 (+ 211us) server_negotiation.cc:407] Connection header received
0810 15:15:48.972097 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:48.972099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:48.972112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:48.972164 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:48.973122 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.973617 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.974588 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.974781 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.974846 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:48.976887 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:48.976889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:48.976890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:48.976897 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:48.978426 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.978835 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.978836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.978837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.978906 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:48.979125 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:48.979126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:48.979126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:48.979243 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:48.979328 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:48.979614 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 15:15:48.979699 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:15:48.979954 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:48.971245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42867 (local address 127.29.230.2:48695)
0810 15:15:48.971458 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:48.971496 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:48.971995 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:48.972234 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:48.972238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:48.972808 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:48.973004 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:48.973010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.973703 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.973704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:48.974498 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:48.974501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:48.974859 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:48.974860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:48.974978 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:48.975478 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:48.975486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:48.976795 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:48.978520 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:48.978527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:48.978530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:48.978769 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:48.978961 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:48.978961 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:48.978961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:48.979069 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:48.979319 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:48.979326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:48.979494 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 15:15:48.979586 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:15:48.979745 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260810 15:15:48.980432 31271 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42867
I20260810 15:15:48.980543 31271 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:48.980733 31271 heartbeater.cc:511] Master 127.29.230.62:42867 requested a full tablet report, sending...
I20260810 15:15:48.981096 30708 ts_manager.cc:194] Registered new tserver with Master: 59376d7f4a014ea19d02cf0b0d233398 (127.29.230.2:33563)
I20260810 15:15:48.981459 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:48695
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:15:49.006745 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:49.090063 31278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:49.090270 31278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:49.090294 31278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:49.091414 31278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:49.091449 31278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:49.091473 31278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:15:49.092973 31278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42867
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:49.093185 31278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:49.093374 31278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:49.098708 31290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:49.098876 31278 server_base.cc:1034] running on GCE node
W20260810 15:15:49.098708 31286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:49.098789 31287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:49.099169 31278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:49.099437 31278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:49.100584 31278 hybrid_clock.cc:648] HybridClock initialized: now 1786374949100571 us; error 30 us; skew 500 ppm
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:49.103174 31278 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:15:49.103670 31278 webserver.cc:460] Webserver started at http://127.29.230.3:34027/ using document root <none> and password file <none>
I20260810 15:15:49.103895 31278 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:49.103943 31278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:49.104069 31278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:49.105178 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "c5d6f42bb76f462badd0b070799af1db"
format_stamp: "Formatted at 2026-08-10 15:15:49 on dist-test-slave-gjrz"
server_key: "d342900d2e1f8ff8a73945aae5799a89"
server_key_iv: "f213d8e2e65729a2761ed663a56730bb"
server_key_version: "encryptionkey@0"
I20260810 15:15:49.105521 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "c5d6f42bb76f462badd0b070799af1db"
format_stamp: "Formatted at 2026-08-10 15:15:49 on dist-test-slave-gjrz"
server_key: "d342900d2e1f8ff8a73945aae5799a89"
server_key_iv: "f213d8e2e65729a2761ed663a56730bb"
server_key_version: "encryptionkey@0"
I20260810 15:15:49.106819 31278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:15:49.107575 31300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:49.107729 31278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:15:49.107815 31278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "c5d6f42bb76f462badd0b070799af1db"
format_stamp: "Formatted at 2026-08-10 15:15:49 on dist-test-slave-gjrz"
server_key: "d342900d2e1f8ff8a73945aae5799a89"
server_key_iv: "f213d8e2e65729a2761ed663a56730bb"
server_key_version: "encryptionkey@0"
I20260810 15:15:49.107880 31278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:49.122849 31278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:49.124987 31278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:49.125128 31278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:49.125356 31278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:49.125686 31278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:49.125746 31278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:49.125785 31278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:49.125801 31278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:49.131919 31278 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:43469
I20260810 15:15:49.131974 31500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:43469 every 8 connection(s)
I20260810 15:15:49.132296 31278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:15:49.132481 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31278
I20260810 15:15:49.132577 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:15:49.132769 30616 external_mini_cluster.cc:1428] Setting key f968ba270435a5d28d136f80cf53b0a3
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:49.141475 30780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:49.133139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:55349 (local address 127.29.230.62:42867)
0810 15:15:49.133186 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:49.133188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:49.133446 (+ 258us) server_negotiation.cc:407] Connection header received
0810 15:15:49.134051 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:49.134052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:49.134063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:49.134113 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:49.135050 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.135598 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.136579 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.136774 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.136829 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:49.138937 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:49.138939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:49.138939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:49.138947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:49.140215 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.140574 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.140575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.140576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.140631 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.140847 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.140848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.140848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.140959 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:49.141018 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:49.141293 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 15:15:49.141387 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:15:49.141644 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:49.133123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42867 (local address 127.29.230.3:55349)
0810 15:15:49.133306 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:49.133357 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:49.133949 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:49.134188 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:49.134192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:49.134760 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:49.134960 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:49.134965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.135693 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.135694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:49.136500 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:49.136502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.136843 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.136844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:49.136977 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:49.137453 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:49.137462 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:49.138845 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:49.140289 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:49.140296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:49.140298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:49.140508 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:49.140685 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:49.140686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:49.140687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:49.140791 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:49.141016 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:49.141022 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:49.141171 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:15:49.141262 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:15:49.141419 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260810 15:15:49.142125 31501 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42867
I20260810 15:15:49.142220 31501 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:49.142393 31501 heartbeater.cc:511] Master 127.29.230.62:42867 requested a full tablet report, sending...
I20260810 15:15:49.142753 30708 ts_manager.cc:194] Registered new tserver with Master: c5d6f42bb76f462badd0b070799af1db (127.29.230.3:43469)
I20260810 15:15:49.143123 30708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:55349
I20260810 15:15:49.145032 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:49.170055 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:49.161718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.29.230.1:35231)
0810 15:15:49.161950 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:49.161956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:49.161969 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:15:49.162019 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:49.162022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:49.162040 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:49.162114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:49.162813 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.163979 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.164844 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.165255 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.165316 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:49.167390 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:49.167398 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:49.167399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:49.167405 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:49.168911 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.169319 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.169323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.169324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.169389 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.169580 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.169581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.169581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.169715 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:49.169834 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:49.169907 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 15:15:49.169975 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
W20260810 15:15:49.170531 30897 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33720
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[30630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:49.193336 31512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:49.186497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.29.230.1:35231)
0810 15:15:49.186582 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:49.186584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:49.186624 (+ 40us) server_negotiation.cc:407] Connection header received
0810 15:15:49.186681 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:49.186682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:49.186694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:49.186742 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:49.187413 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.187916 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.188655 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:49.188852 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:49.188900 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:49.191069 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:49.191071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:49.191071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:49.191080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:49.192316 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.192650 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.192651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.192651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.192710 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:49.192933 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:49.192934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:49.192934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:49.193034 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:49.193123 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:49.193187 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 15:15:49.193260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 15:15:49.193964 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30821
I20260810 15:15:49.207095 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31055
I20260810 15:15:49.254335 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31278
I20260810 15:15:49.274399 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30649
2026-08-10T15:15:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2122 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[31524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[31524](info): set up 2 sockets
Aug 10 15:15:49 dist-test-slave-gjrz krb5kdc[31524](info): commencing operation
krb5kdc: starting...
W20260810 15:15:50.469348 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:15:50 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:15:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:15:50.590471 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:38993
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:38993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:50.672528 31542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:50.672756 31542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:50.672780 31542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:50.673987 31542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:15:50.674033 31542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:50.674046 31542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:50.674057 31542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:15:50.674068 31542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:15:50.675654 31542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:38993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:38993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:50.675882 31542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:50.676127 31542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:50.680858 31553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:50.680866 31555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:50.680987 31542 server_base.cc:1034] running on GCE node
W20260810 15:15:50.680881 31557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:50.681342 31542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:50.681633 31542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:50.682776 31542 hybrid_clock.cc:648] HybridClock initialized: now 1786374950682760 us; error 29 us; skew 500 ppm
Aug 10 15:15:50 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:50.689641 31542 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:15:50.690070 31542 webserver.cc:460] Webserver started at http://127.29.230.62:33095/ using document root <none> and password file <none>
I20260810 15:15:50.690326 31542 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:50.690378 31542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:50.690503 31542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:50.691642 31542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "9670e96a6d5c4c369493eefd7f06ede2"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "17544b1bc9f0fdd9814ed453319eec0c"
server_key_iv: "f452969245203f9643894359c380f7e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.691980 31542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "9670e96a6d5c4c369493eefd7f06ede2"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "17544b1bc9f0fdd9814ed453319eec0c"
server_key_iv: "f452969245203f9643894359c380f7e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.693243 31542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:50.693951 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:50.694124 31542 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:50.694211 31542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "9670e96a6d5c4c369493eefd7f06ede2"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "17544b1bc9f0fdd9814ed453319eec0c"
server_key_iv: "f452969245203f9643894359c380f7e5"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.694271 31542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:50.704501 31542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:50.706272 31542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:50.706404 31542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:50.710300 31542 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:38993
I20260810 15:15:50.710333 31656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:38993 every 8 connection(s)
I20260810 15:15:50.710657 31542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:15:50.711552 31657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:50.713864 31657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2: Bootstrap starting.
I20260810 15:15:50.714426 31657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:50.714689 31657 log.cc:824] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:50.715474 31657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2: No bootstrap required, opened a new log
I20260810 15:15:50.715677 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31542
I20260810 15:15:50.715752 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:15:50.715889 30616 external_mini_cluster.cc:1428] Setting key 3d7e6131e3dad7f3ab64fe791bb4c626
I20260810 15:15:50.717442 31657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } }
I20260810 15:15:50.717599 31657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:50.717623 31657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9670e96a6d5c4c369493eefd7f06ede2, State: Initialized, Role: FOLLOWER
I20260810 15:15:50.717736 31657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } }
I20260810 15:15:50.717810 31657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:15:50.717844 31657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:15:50.717891 31657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:50.718634 31657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } }
I20260810 15:15:50.718748 31657 leader_election.cc:302] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9670e96a6d5c4c369493eefd7f06ede2; no voters:
I20260810 15:15:50.718912 31657 leader_election.cc:288] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:15:50.718964 31665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:50.719154 31665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [term 1 LEADER]: Becoming Leader. State: Replica: 9670e96a6d5c4c369493eefd7f06ede2, State: Running, Role: LEADER
I20260810 15:15:50.719210 31657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:15:50.719304 31665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } }
I20260810 15:15:50.719645 31665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9670e96a6d5c4c369493eefd7f06ede2. Latest consensus state: current_term: 1 leader_uuid: "9670e96a6d5c4c369493eefd7f06ede2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } } }
I20260810 15:15:50.719867 31665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:15:50.719803 31666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9670e96a6d5c4c369493eefd7f06ede2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9670e96a6d5c4c369493eefd7f06ede2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38993 } } }
I20260810 15:15:50.720273 31666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:15:50.720507 31684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:15:50.721134 31684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:15:50.723398 31684 catalog_manager.cc:1347] Generated new cluster ID: 5626f02c24294e0398c4898a39ea06e0
I20260810 15:15:50.723450 31684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:15:50 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:50.728384 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:50.716385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.29.230.62:38993)
0810 15:15:50.716586 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:50.716594 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:50.716619 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:15:50.717045 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:50.717052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:50.717106 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:50.717230 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:50.717887 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.720057 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.721150 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.721741 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.721852 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:50.724584 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:50.724588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:50.724588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:50.724600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:50.726589 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:50.727026 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:50.727028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:50.727029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:50.727135 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:50.727368 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:50.727369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:50.727369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:50.727662 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:50.727813 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:50.727942 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 15:15:50.728185 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260810 15:15:50.737332 31684 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:15:50.738092 31684 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:15:50.749809 31684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9670e96a6d5c4c369493eefd7f06ede2: Generated new TSK 0
I20260810 15:15:50.750039 31684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:15:50.782712 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:50.864341 31703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:50.864538 31703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:50.864570 31703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:50.865691 31703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:50.865731 31703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:50.865767 31703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:15:50.867228 31703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:50.867475 31703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:50.867687 31703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:50.872327 31712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:50.872327 31716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:50.872381 31713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:50.872462 31703 server_base.cc:1034] running on GCE node
I20260810 15:15:50.872792 31703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:50.872994 31703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:50.874153 31703 hybrid_clock.cc:648] HybridClock initialized: now 1786374950874136 us; error 40 us; skew 500 ppm
Aug 10 15:15:50 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:50.876602 31703 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:15:50.877000 31703 webserver.cc:460] Webserver started at http://127.29.230.1:35639/ using document root <none> and password file <none>
I20260810 15:15:50.877249 31703 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:50.877298 31703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:50.877417 31703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:50.878433 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "afc7ad0c0f6f4660975234e1db9bb90e"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "f1c146e79d43e11c925461a3422fc298"
server_key_iv: "7e62afa27a5b9ae5d912d3404202a078"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.878767 31703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "afc7ad0c0f6f4660975234e1db9bb90e"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "f1c146e79d43e11c925461a3422fc298"
server_key_iv: "7e62afa27a5b9ae5d912d3404202a078"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.880122 31703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:15:50.880844 31734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:50.881078 31703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:50.881151 31703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "afc7ad0c0f6f4660975234e1db9bb90e"
format_stamp: "Formatted at 2026-08-10 15:15:50 on dist-test-slave-gjrz"
server_key: "f1c146e79d43e11c925461a3422fc298"
server_key_iv: "7e62afa27a5b9ae5d912d3404202a078"
server_key_version: "encryptionkey@0"
I20260810 15:15:50.881215 31703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:50.898286 31703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:50.900380 31703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:50.900521 31703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:50.900722 31703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:50.901036 31703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:50.901074 31703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:50.901104 31703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:50.901129 31703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:50.906678 31703 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:32793
I20260810 15:15:50.906719 31943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:32793 every 8 connection(s)
I20260810 15:15:50.907045 31703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:15:50.908385 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31703
I20260810 15:15:50.908468 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:15:50.908620 30616 external_mini_cluster.cc:1428] Setting key dbeb6ccdb769cb36b87e4b896805e8b2
Aug 10 15:15:50 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:50.917619 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:50.907884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:40321 (local address 127.29.230.62:38993)
0810 15:15:50.907962 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:50.907965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:50.908122 (+ 157us) server_negotiation.cc:407] Connection header received
0810 15:15:50.909304 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:50.909305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:50.909317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:50.909370 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:50.909995 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.910670 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.912132 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.912353 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.912415 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:50.914895 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:50.914897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:50.914897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:50.914904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:50.916199 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:50.916668 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:50.916669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:50.916670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:50.916724 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:50.916972 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:50.916973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:50.916973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:50.917080 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:50.917128 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:50.917465 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 15:15:50.917553 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:15:50.917919 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:50.907835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38993 (local address 127.29.230.1:40321)
0810 15:15:50.908005 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:50.908044 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:50.909209 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:50.909443 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:50.909447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:50.909706 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:50.909902 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:50.909905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.910772 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.910773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:50.912033 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:50.912036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:50.912467 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:50.912468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:50.912611 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:50.913138 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:50.913148 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:50.914803 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:50.916277 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:50.916285 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:50.916288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:50.916586 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:50.916778 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:50.916779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:50.916780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:50.916913 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:50.917139 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:50.917145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:50.917314 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 15:15:50.917454 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 15:15:50.917620 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260810 15:15:50.918610 31944 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38993
I20260810 15:15:50.918711 31944 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:50.918965 31944 heartbeater.cc:511] Master 127.29.230.62:38993 requested a full tablet report, sending...
I20260810 15:15:50.919544 31603 ts_manager.cc:194] Registered new tserver with Master: afc7ad0c0f6f4660975234e1db9bb90e (127.29.230.1:32793)
I20260810 15:15:50.920287 31603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:40321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:15:50.935876 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:51.017949 31953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:51.018180 31953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:51.018208 31953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:51.019495 31953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:51.019539 31953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:51.019565 31953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:15:51.021211 31953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:51.021442 31953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:51.021665 31953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:51.026623 31963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:51.026613 31959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:51.026612 31958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:51.026925 31953 server_base.cc:1034] running on GCE node
I20260810 15:15:51.027177 31953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:51.027448 31953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:51.028609 31953 hybrid_clock.cc:648] HybridClock initialized: now 1786374951028581 us; error 36 us; skew 500 ppm
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:51.031102 31953 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:15:51.031574 31953 webserver.cc:460] Webserver started at http://127.29.230.2:42581/ using document root <none> and password file <none>
I20260810 15:15:51.031790 31953 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:51.031838 31953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:51.031963 31953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:51.033021 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "1bbc891197b741a9a33fdda1d08e3a46"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "d9074e55e14be1df9437679612bd4f0c"
server_key_iv: "d4669458c6382f4f7519dce362cd535e"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.033370 31953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "1bbc891197b741a9a33fdda1d08e3a46"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "d9074e55e14be1df9437679612bd4f0c"
server_key_iv: "d4669458c6382f4f7519dce362cd535e"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.034674 31953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:15:51.035477 31974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.035635 31953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:51.035722 31953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "1bbc891197b741a9a33fdda1d08e3a46"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "d9074e55e14be1df9437679612bd4f0c"
server_key_iv: "d4669458c6382f4f7519dce362cd535e"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.035782 31953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:51.055905 31953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:51.058123 31953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:51.058279 31953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:51.058517 31953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:51.058826 31953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:51.058859 31953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.058888 31953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:51.058912 31953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.064762 31953 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:45801
I20260810 15:15:51.064821 32168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:45801 every 8 connection(s)
I20260810 15:15:51.065140 31953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:15:51.071422 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31953
I20260810 15:15:51.071507 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:51.071664 30616 external_mini_cluster.cc:1428] Setting key f32d647fcb61cbf5be1d4dbc38976526
I20260810 15:15:51.075052 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.065864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:41233 (local address 127.29.230.62:38993)
0810 15:15:51.065916 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.065919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.066140 (+ 221us) server_negotiation.cc:407] Connection header received
0810 15:15:51.067089 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.067090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.067102 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.067160 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.067840 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.068360 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.069515 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.069698 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.069747 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.072149 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.072151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.072151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.072159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.073605 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.073993 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.073994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.073995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.074061 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.074338 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.074339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.074339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.074507 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.074573 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.074857 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.074959 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:15:51.075292 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.065837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38993 (local address 127.29.230.2:41233)
0810 15:15:51.066016 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.066057 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.066973 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.067228 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.067232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.067563 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:51.067766 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.067770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.068466 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.068466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.069444 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.069447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.069753 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.069753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.069849 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.070338 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:51.070347 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:51.072058 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:51.073681 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:51.073688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:51.073691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:51.073921 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:51.074119 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:51.074120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:51.074120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:51.074273 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:51.074577 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:51.074584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:51.074732 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.074828 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.075038 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260810 15:15:51.075928 32169 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38993
I20260810 15:15:51.076025 32169 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:51.076211 32169 heartbeater.cc:511] Master 127.29.230.62:38993 requested a full tablet report, sending...
I20260810 15:15:51.076606 31603 ts_manager.cc:194] Registered new tserver with Master: 1bbc891197b741a9a33fdda1d08e3a46 (127.29.230.2:45801)
I20260810 15:15:51.077044 31603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:41233
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:15:51.099946 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:51.182699 32175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:51.182906 32175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:51.182942 32175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:51.184221 32175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:51.184270 32175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:51.184309 32175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:15:51.185781 32175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38993
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:51.186030 32175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:51.186249 32175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:51.191164 32188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:51.191211 32175 server_base.cc:1034] running on GCE node
W20260810 15:15:51.191169 32193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:51.191164 32189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:51.191609 32175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:51.191844 32175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:51.193028 32175 hybrid_clock.cc:648] HybridClock initialized: now 1786374951192998 us; error 46 us; skew 500 ppm
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:51.195560 32175 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:15:51.195989 32175 webserver.cc:460] Webserver started at http://127.29.230.3:45025/ using document root <none> and password file <none>
I20260810 15:15:51.196195 32175 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:51.196240 32175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:51.196344 32175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:51.197395 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "3ebb4b42b000cdaf48b62720655a902a"
server_key_iv: "f1d1e023b476c518314e4b32cb3e1d5b"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.197718 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "3ebb4b42b000cdaf48b62720655a902a"
server_key_iv: "f1d1e023b476c518314e4b32cb3e1d5b"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.198940 32175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:15:51.199720 32207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.199877 32175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.199962 32175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a"
format_stamp: "Formatted at 2026-08-10 15:15:51 on dist-test-slave-gjrz"
server_key: "3ebb4b42b000cdaf48b62720655a902a"
server_key_iv: "f1d1e023b476c518314e4b32cb3e1d5b"
server_key_version: "encryptionkey@0"
I20260810 15:15:51.200022 32175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:51.211491 32175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:51.213583 32175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:51.213711 32175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:51.213908 32175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:15:51.214210 32175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:51.214241 32175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.214260 32175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:51.214272 32175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:51.219928 32175 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:37761
I20260810 15:15:51.219981 32396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:37761 every 8 connection(s)
I20260810 15:15:51.220304 32175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:15:51.225679 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32175
I20260810 15:15:51.225773 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:15:51.225924 30616 external_mini_cluster.cc:1428] Setting key 149161689a2ae785629c0d0a4f70ba00
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:51.229568 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.221188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:35397 (local address 127.29.230.62:38993)
0810 15:15:51.221246 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.221248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.221438 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:15:51.222424 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.222425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.222446 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.222494 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.223144 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.223705 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.224704 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.224903 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.224958 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.227232 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.227234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.227234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.227240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.228453 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.228808 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.228809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.228809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.228861 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.229058 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.229058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.229059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.229169 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.229219 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.229439 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.229506 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:15:51.229719 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.221175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38993 (local address 127.29.230.3:35397)
0810 15:15:51.221321 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.221357 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.222318 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.222581 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.222585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.222838 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:51.223055 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.223059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.223789 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.223790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.224620 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.224623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.224975 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.224976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.225111 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.225556 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:51.225564 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:51.227137 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:51.228528 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:51.228534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:51.228537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:51.228745 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:51.228901 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:51.228901 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:51.228901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:51.229008 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:51.229216 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:51.229219 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:51.229331 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.229405 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.229526 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260810 15:15:51.230140 32397 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38993
I20260810 15:15:51.230228 32397 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:51.230376 32397 heartbeater.cc:511] Master 127.29.230.62:38993 requested a full tablet report, sending...
I20260810 15:15:51.230728 31603 ts_manager.cc:194] Registered new tserver with Master: b3b1fa58124e46fe9663d8dcaf0f3c2a (127.29.230.3:37761)
I20260810 15:15:51.231173 31603 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:35397
I20260810 15:15:51.237613 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:15:51.245345 31664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.239661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.29.230.62:38993)
0810 15:15:51.239714 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.239716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.239847 (+ 131us) server_negotiation.cc:407] Connection header received
0810 15:15:51.239889 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.239889 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.239898 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.239947 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.240466 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.240942 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.241627 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.241814 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.241865 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.242982 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.242983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.242983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.242989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.244226 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.244550 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.244550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.244551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.244600 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.244769 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.244771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.244771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.244939 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.245054 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.245208 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.245272 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:15:51.248306 31603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:15:51.250201 31603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:51.257169 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46784 (local address 127.29.230.2:45801)
0810 15:15:51.253307 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.253311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.253321 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:15:51.253360 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.253362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.253378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.253469 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:51.253770 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.254900 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.256105 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.256673 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.256745 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.256805 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.257063 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.257103 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260810 15:15:51.257172 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:45801 (local address 127.0.0.1:46784)
0810 15:15:51.253166 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.253205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.253265 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.253569 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.253573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.253598 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:51.253675 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.253677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.255023 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.255024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.256009 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.256012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.256840 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.256841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.256937 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.256950 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.257021 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.257088 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":106,"thread_start_us":64,"threads_started":1}
I20260810 15:15:51.257699 32421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32793 (local address 127.0.0.1:52092)
0810 15:15:51.253651 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.253661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.253711 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.253958 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.253959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.253961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:51.253993 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.253993 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.254963 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.254964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.256385 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.256389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.257470 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.257472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.257536 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.257542 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.257595 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.257642 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":71,"thread_start_us":59,"threads_started":1}
I20260810 15:15:51.257755 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.29.230.1:32793)
0810 15:15:51.253824 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.253830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.253839 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:51.253878 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.253880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.253895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.253964 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:51.254044 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.254867 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.256478 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.257195 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.257261 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.257324 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.257629 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.257671 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260810 15:15:51.258301 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60148 (local address 127.29.230.3:37761)
0810 15:15:51.253651 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.253655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.253665 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:15:51.253701 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.253703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.253718 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.253794 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:51.253925 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.254960 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.257364 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.258005 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.258063 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.258102 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.258214 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.258243 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260810 15:15:51.258309 32420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.253264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:37761 (local address 127.0.0.1:60148)
0810 15:15:51.253396 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.253408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.253452 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.253795 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.253796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.253798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:51.253856 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.253857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.255056 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.255057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.256385 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.256389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.258071 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.258072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.258136 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.258140 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.258204 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.258244 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":94,"thread_start_us":80,"threads_started":1}
I20260810 15:15:51.258370 32057 tablet_service.cc:1467] Processing CreateTablet for tablet 5421911268214ebc817b487a59694254 (DEFAULT_TABLE table=test-table [id=67e83f2a0fec4d6ca803fd09b18c0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:51.258780 32057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5421911268214ebc817b487a59694254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:51.261554 32426 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Bootstrap starting.
I20260810 15:15:51.262308 32426 tablet_bootstrap.cc:654] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:51.262408 32289 tablet_service.cc:1467] Processing CreateTablet for tablet 5421911268214ebc817b487a59694254 (DEFAULT_TABLE table=test-table [id=67e83f2a0fec4d6ca803fd09b18c0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:51.262684 32289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5421911268214ebc817b487a59694254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:51.262580 31822 tablet_service.cc:1467] Processing CreateTablet for tablet 5421911268214ebc817b487a59694254 (DEFAULT_TABLE table=test-table [id=67e83f2a0fec4d6ca803fd09b18c0d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:51.262946 31822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5421911268214ebc817b487a59694254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:51.263108 32426 log.cc:824] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:51.264034 32426 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: No bootstrap required, opened a new log
I20260810 15:15:51.264115 32426 ts_tablet_manager.cc:1397] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 15:15:51.265417 32429 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: Bootstrap starting.
I20260810 15:15:51.266000 32429 tablet_bootstrap.cc:654] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:51.266129 32430 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: Bootstrap starting.
I20260810 15:15:51.266057 32426 raft_consensus.cc:348] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.266251 32426 raft_consensus.cc:374] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:51.266304 32426 raft_consensus.cc:729] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbc891197b741a9a33fdda1d08e3a46, State: Initialized, Role: FOLLOWER
I20260810 15:15:51.266450 32426 consensus_queue.cc:260] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.266716 32426 ts_tablet_manager.cc:1428] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:15:51.266774 32430 tablet_bootstrap.cc:654] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:51.266779 32429 log.cc:824] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:51.266913 32169 heartbeater.cc:503] Master 127.29.230.62:38993 was elected leader, sending a full tablet report...
I20260810 15:15:51.267304 32430 log.cc:824] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:51.267879 32429 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: No bootstrap required, opened a new log
I20260810 15:15:51.267951 32429 ts_tablet_manager.cc:1397] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 15:15:51.269361 32429 raft_consensus.cc:348] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.269493 32429 raft_consensus.cc:374] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:51.269524 32429 raft_consensus.cc:729] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b1fa58124e46fe9663d8dcaf0f3c2a, State: Initialized, Role: FOLLOWER
I20260810 15:15:51.269641 32429 consensus_queue.cc:260] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.269882 32429 ts_tablet_manager.cc:1428] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:51.269906 32397 heartbeater.cc:503] Master 127.29.230.62:38993 was elected leader, sending a full tablet report...
I20260810 15:15:51.270458 32430 tablet_bootstrap.cc:492] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: No bootstrap required, opened a new log
I20260810 15:15:51.270529 32430 ts_tablet_manager.cc:1397] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 15:15:51.271961 32430 raft_consensus.cc:348] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.272109 32430 raft_consensus.cc:374] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:51.272132 32430 raft_consensus.cc:729] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afc7ad0c0f6f4660975234e1db9bb90e, State: Initialized, Role: FOLLOWER
I20260810 15:15:51.272217 32430 consensus_queue.cc:260] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.272503 32430 ts_tablet_manager.cc:1428] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:51.272500 31944 heartbeater.cc:503] Master 127.29.230.62:38993 was elected leader, sending a full tablet report...
I20260810 15:15:51.279488 32435 raft_consensus.cc:482] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:51.279592 32435 raft_consensus.cc:504] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.280131 32435 leader_election.cc:288] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bbc891197b741a9a33fdda1d08e3a46 (127.29.230.2:45801), b3b1fa58124e46fe9663d8dcaf0f3c2a (127.29.230.3:37761)
I20260810 15:15:51.283056 32419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.280377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:56477 (local address 127.29.230.2:45801)
0810 15:15:51.280440 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.280442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.280474 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:15:51.280503 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.280504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.280515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.280566 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:51.280716 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.281251 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282323 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.282724 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.282785 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.282936 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.282978 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:15:51.283082 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.280505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:38767 (local address 127.29.230.3:37761)
0810 15:15:51.280560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.280561 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.280672 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:15:51.280722 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.280723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.280735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.280776 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:51.280919 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.281443 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282395 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.282772 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282841 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.282849 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.282985 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.283023 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:15:51.283083 31946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.280305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:45801 (local address 127.29.230.1:56477)
0810 15:15:51.280366 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.280380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.280435 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.280573 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.280574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.280576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:51.280646 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.280647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.281327 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.281329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.282226 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.282228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.282793 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.282872 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.282875 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.282923 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.282956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260810 15:15:51.283090 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.280413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:37761 (local address 127.29.230.1:38767)
0810 15:15:51.280612 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:51.280622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:51.280673 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:51.280788 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:51.280788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:51.280790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:51.280857 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.280859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.281535 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.281536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.282310 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:51.282313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.282849 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.282850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:51.282914 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.282919 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:15:51.282972 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:15:51.283009 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260810 15:15:51.283610 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5421911268214ebc817b487a59694254" candidate_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbc891197b741a9a33fdda1d08e3a46" is_pre_election: true
I20260810 15:15:51.283615 32320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5421911268214ebc817b487a59694254" candidate_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" is_pre_election: true
I20260810 15:15:51.283790 32320 raft_consensus.cc:2454] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc7ad0c0f6f4660975234e1db9bb90e in term 0.
I20260810 15:15:51.283792 32093 raft_consensus.cc:2454] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afc7ad0c0f6f4660975234e1db9bb90e in term 0.
I20260810 15:15:51.284163 31742 leader_election.cc:302] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbc891197b741a9a33fdda1d08e3a46, afc7ad0c0f6f4660975234e1db9bb90e; no voters:
I20260810 15:15:51.284286 32435 raft_consensus.cc:2781] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:51.284360 32435 raft_consensus.cc:482] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:51.284394 32435 raft_consensus.cc:3037] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:51.285142 32435 raft_consensus.cc:504] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.285311 32435 leader_election.cc:288] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [CANDIDATE]: Term 1 election: Requested vote from peers 1bbc891197b741a9a33fdda1d08e3a46 (127.29.230.2:45801), b3b1fa58124e46fe9663d8dcaf0f3c2a (127.29.230.3:37761)
I20260810 15:15:51.285495 32093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5421911268214ebc817b487a59694254" candidate_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbc891197b741a9a33fdda1d08e3a46"
I20260810 15:15:51.285513 32320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5421911268214ebc817b487a59694254" candidate_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a"
I20260810 15:15:51.285575 32093 raft_consensus.cc:3037] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:51.285579 32320 raft_consensus.cc:3037] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:51.286288 32320 raft_consensus.cc:2454] T 5421911268214ebc817b487a59694254 P b3b1fa58124e46fe9663d8dcaf0f3c2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc7ad0c0f6f4660975234e1db9bb90e in term 1.
I20260810 15:15:51.286302 32093 raft_consensus.cc:2454] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afc7ad0c0f6f4660975234e1db9bb90e in term 1.
I20260810 15:15:51.286444 31742 leader_election.cc:302] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbc891197b741a9a33fdda1d08e3a46, afc7ad0c0f6f4660975234e1db9bb90e; no voters:
I20260810 15:15:51.286549 32435 raft_consensus.cc:2781] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:51.286710 32435 raft_consensus.cc:686] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [term 1 LEADER]: Becoming Leader. State: Replica: afc7ad0c0f6f4660975234e1db9bb90e, State: Running, Role: LEADER
I20260810 15:15:51.286793 32435 consensus_queue.cc:237] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } }
I20260810 15:15:51.287761 31603 catalog_manager.cc:5495] T 5421911268214ebc817b487a59694254 P afc7ad0c0f6f4660975234e1db9bb90e reported cstate change: term changed from 0 to 1, leader changed from <none> to afc7ad0c0f6f4660975234e1db9bb90e (127.29.230.1). New cstate: current_term: 1 leader_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbc891197b741a9a33fdda1d08e3a46" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 45801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b1fa58124e46fe9663d8dcaf0f3c2a" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 37761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afc7ad0c0f6f4660975234e1db9bb90e" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32793 } health_report { overall_health: HEALTHY } } }
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:51.299926 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.292705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.29.230.1:32793)
0810 15:15:51.292756 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.292758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.292836 (+ 78us) server_negotiation.cc:407] Connection header received
0810 15:15:51.292872 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.292873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.292882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.292929 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.293409 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.293857 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.294618 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.294809 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.294863 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.297237 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.297246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.297246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.297253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.298808 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.299142 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.299145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.299145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.299203 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.299433 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.299433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.299434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.299578 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.299688 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.299795 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.299858 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260810 15:15:51.315723 32171 tablet.cc:2369] T 5421911268214ebc817b487a59694254 P 1bbc891197b741a9a33fdda1d08e3a46: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:51.324064 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.316992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.29.230.1:32793)
0810 15:15:51.317047 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.317049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.317160 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:15:51.317209 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.317210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.317219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.317276 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.317834 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.318373 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.319171 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.319417 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.319480 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.321726 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.321727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.321728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.321734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.322930 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.323301 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.323302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.323302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.323358 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.323526 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.323527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.323527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.323626 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.323713 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.323885 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.323964 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[31524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:51.346555 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:51.340124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.29.230.1:32793)
0810 15:15:51.340187 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:51.340189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:51.340265 (+ 76us) server_negotiation.cc:407] Connection header received
0810 15:15:51.340320 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:51.340320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:51.340330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:51.340378 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:51.340856 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.341323 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.342044 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:51.342223 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:51.342268 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:51.344371 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:51.344372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:51.344373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:51.344379 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:51.345616 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.345934 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.345935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.345935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.345992 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:51.346193 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:51.346193 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:51.346194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:51.346282 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:51.346356 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:51.346427 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 15:15:51.346487 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 15:15:51.347352 31788 tablet_service.cc:2908] Rejecting scan request for tablet 5421911268214ebc817b487a59694254: Uninitialized: clean time has not yet been initialized
I20260810 15:15:51.348578 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31703
I20260810 15:15:51.363371 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31953
I20260810 15:15:51.372774 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32175
I20260810 15:15:51.389855 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31542
2026-08-10T15:15:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2111 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[32460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[32460](info): set up 2 sockets
Aug 10 15:15:51 dist-test-slave-gjrz krb5kdc[32460](info): commencing operation
krb5kdc: starting...
W20260810 15:15:52.583216 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:15:52 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:15:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:15:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:15:52.718675 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:43393
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:43393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:52.801549 32477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:52.801757 32477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:52.801780 32477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:52.802948 32477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:15:52.802989 32477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:15:52.803001 32477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:52.803012 32477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:52.803025 32477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:15:52.803036 32477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:15:52.804657 32477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:43393
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:43393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:52.804898 32477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:52.805127 32477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:52.810160 32503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:52.810160 32487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:52.810228 32495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:52.810428 32477 server_base.cc:1034] running on GCE node
I20260810 15:15:52.810673 32477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:52.810981 32477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:52.812134 32477 hybrid_clock.cc:648] HybridClock initialized: now 1786374952812119 us; error 27 us; skew 500 ppm
Aug 10 15:15:52 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374952, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:52.814756 32477 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:15:52.815193 32477 webserver.cc:460] Webserver started at http://127.29.230.62:38151/ using document root <none> and password file <none>
I20260810 15:15:52.815423 32477 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:52.815471 32477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:52.815600 32477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:52.816694 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "c58ade0936f84f14890562383e2afd15"
format_stamp: "Formatted at 2026-08-10 15:15:52 on dist-test-slave-gjrz"
server_key: "6f6da35ae6292aa8c35c92dab24a68f6"
server_key_iv: "c197a9fb9ad7fc75d859b3377f272681"
server_key_version: "encryptionkey@0"
I20260810 15:15:52.817165 32477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "c58ade0936f84f14890562383e2afd15"
format_stamp: "Formatted at 2026-08-10 15:15:52 on dist-test-slave-gjrz"
server_key: "6f6da35ae6292aa8c35c92dab24a68f6"
server_key_iv: "c197a9fb9ad7fc75d859b3377f272681"
server_key_version: "encryptionkey@0"
I20260810 15:15:52.818535 32477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:15:52.819212 32519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:52.819401 32477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:52.819499 32477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "c58ade0936f84f14890562383e2afd15"
format_stamp: "Formatted at 2026-08-10 15:15:52 on dist-test-slave-gjrz"
server_key: "6f6da35ae6292aa8c35c92dab24a68f6"
server_key_iv: "c197a9fb9ad7fc75d859b3377f272681"
server_key_version: "encryptionkey@0"
I20260810 15:15:52.819566 32477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:52.852329 32477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:52.854609 32477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:52.854728 32477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:52.859503 32477 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:43393
I20260810 15:15:52.859586 32628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:43393 every 8 connection(s)
I20260810 15:15:52.859886 32477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:15:52.860626 32629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:52.862948 32629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Bootstrap starting.
I20260810 15:15:52.863544 32629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:52.863816 32629 log.cc:824] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:52.864552 32629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: No bootstrap required, opened a new log
I20260810 15:15:52.864598 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32477
I20260810 15:15:52.864683 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:15:52.864861 30616 external_mini_cluster.cc:1428] Setting key 45478970cc030082e976b8f0986042dc
I20260810 15:15:52.867127 32629 raft_consensus.cc:348] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } }
I20260810 15:15:52.867309 32629 raft_consensus.cc:374] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:52.867357 32629 raft_consensus.cc:729] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c58ade0936f84f14890562383e2afd15, State: Initialized, Role: FOLLOWER
I20260810 15:15:52.867492 32629 consensus_queue.cc:260] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } }
I20260810 15:15:52.867581 32629 raft_consensus.cc:388] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:15:52.867614 32629 raft_consensus.cc:482] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:15:52.867662 32629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:52.868310 32629 raft_consensus.cc:504] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } }
I20260810 15:15:52.868402 32629 leader_election.cc:302] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c58ade0936f84f14890562383e2afd15; no voters:
I20260810 15:15:52.868585 32629 leader_election.cc:288] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:15:52.868624 32639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:52.868736 32639 raft_consensus.cc:686] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [term 1 LEADER]: Becoming Leader. State: Replica: c58ade0936f84f14890562383e2afd15, State: Running, Role: LEADER
I20260810 15:15:52.868818 32639 consensus_queue.cc:237] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } }
I20260810 15:15:52.868851 32629 sys_catalog.cc:564] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:15:52.869194 32640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c58ade0936f84f14890562383e2afd15. Latest consensus state: current_term: 1 leader_uuid: "c58ade0936f84f14890562383e2afd15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } } }
I20260810 15:15:52.869269 32640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [sys.catalog]: This master's current role is: LEADER
I20260810 15:15:52.869516 32644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:15:52.870236 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c58ade0936f84f14890562383e2afd15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c58ade0936f84f14890562383e2afd15" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 43393 } } }
I20260810 15:15:52.870396 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15 [sys.catalog]: This master's current role is: LEADER
I20260810 15:15:52.870513 32644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:15:52.872224 32644 catalog_manager.cc:1347] Generated new cluster ID: 87c549ad84414d6d89a57710cd0c2e96
I20260810 15:15:52.872284 32644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:15:52 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:52.876763 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:52.865393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.29.230.62:43393)
0810 15:15:52.865558 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:52.865566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:52.865594 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:15:52.867098 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:52.867105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:52.867157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:52.867303 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:52.868083 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:52.869474 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:52.870540 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:52.871149 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:52.871281 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:52.873715 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:52.873719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:52.873719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:52.873732 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:52.875227 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:52.875592 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:52.875595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:52.875595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:52.875701 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:52.875879 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:52.875880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:52.875880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:52.876126 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:52.876278 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:52.876386 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 15:15:52.876613 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260810 15:15:52.877574 32644 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:15:52.878058 32644 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:15:52.880584 32644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Generated new TSK 0
I20260810 15:15:52.880733 32644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:15:52.910465 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:43393
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:52.992614 32672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:52.992815 32672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:52.992837 32672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:52.993952 32672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:52.994000 32672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:52.994019 32672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:52.994035 32672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:15:52.995481 32672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:52.995720 32672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:52.995935 32672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:53.000726 32681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:53.000734 32680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:53.000823 32672 server_base.cc:1034] running on GCE node
W20260810 15:15:53.000744 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:53.001157 32672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:53.001394 32672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:53.002527 32672 hybrid_clock.cc:648] HybridClock initialized: now 1786374953002511 us; error 35 us; skew 500 ppm
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:53.005113 32672 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:15:53.005553 32672 webserver.cc:460] Webserver started at http://127.29.230.1:43343/ using document root <none> and password file <none>
I20260810 15:15:53.005775 32672 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:53.005820 32672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:53.005916 32672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:53.006943 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "5f48fdee03730515d02167b3f363af60"
server_key_iv: "baf5707c2ba1ae60f59d47ecc01c3142"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.007254 32672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "5f48fdee03730515d02167b3f363af60"
server_key_iv: "baf5707c2ba1ae60f59d47ecc01c3142"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.008584 32672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:53.009375 32695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.009554 32672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:53.009647 32672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "5f48fdee03730515d02167b3f363af60"
server_key_iv: "baf5707c2ba1ae60f59d47ecc01c3142"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.009708 32672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:53.022454 32672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:53.024665 32672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:53.024803 32672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:53.025314 32672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:53.025353 32672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.025382 32672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:53.025400 32672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:53.032239 32672 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:46247
I20260810 15:15:53.032536 427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:46247 every 8 connection(s)
I20260810 15:15:53.032685 32672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:15:53.034926 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.026023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:33405 (local address 127.29.230.62:43393)
0810 15:15:53.026075 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.026077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.026318 (+ 241us) server_negotiation.cc:407] Connection header received
0810 15:15:53.026846 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.026848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.026867 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.026930 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.027867 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.028414 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.029428 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.029651 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.029710 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.031948 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.031949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.031949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.031958 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.033470 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.033903 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.033904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.033904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.033961 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.034234 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.034235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.034235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.034360 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.034417 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.034767 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.034859 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:15:53.035032 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.026014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.1:33405)
0810 15:15:53.026182 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.026219 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.026734 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.027045 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.027049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.027442 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.027743 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.027750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.028505 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.028506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.029340 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.029342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.029740 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.029742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.029884 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.030396 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.030405 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.031848 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.033558 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.033564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.033566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.033847 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.034013 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.034014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.034014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.034134 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.034456 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.034463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.034597 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.034711 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.034850 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260810 15:15:53.035559 428 heartbeater.cc:347] Connected to a master server at 127.29.230.62:43393
I20260810 15:15:53.035661 428 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:53.035861 428 heartbeater.cc:511] Master 127.29.230.62:43393 requested a full tablet report, sending...
I20260810 15:15:53.036191 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32672
I20260810 15:15:53.036298 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:15:53.036460 30616 external_mini_cluster.cc:1428] Setting key 7562d7c429592f3ffa0b4d99d949854a
I20260810 15:15:53.036473 32549 ts_manager.cc:194] Registered new tserver with Master: edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247)
I20260810 15:15:53.037247 32549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:33405
I20260810 15:15:53.044283 436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.037561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.1:52849)
0810 15:15:53.038138 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.038149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.038205 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.038855 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.038856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.039060 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.039144 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.039145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.039931 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.039932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.040443 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.040444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.040865 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.040865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.040925 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.041372 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.041376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.041666 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.043315 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.043316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.043316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.043483 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.043728 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.043729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.043729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.043781 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.044028 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.044029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.044066 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.044110 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.044197 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":500,"threads_started":1}
I20260810 15:15:53.044317 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.038684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:52849 (local address 127.29.230.62:43393)
0810 15:15:53.038736 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.038737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.038745 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:15:53.038765 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.038766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.038775 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.038819 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.039244 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.039829 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.040525 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.040795 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.040870 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.041750 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.041751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.041751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.041759 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.043201 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.043587 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.043589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.043589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.043660 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.043834 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.043835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.043835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.043972 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.044045 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.044139 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.044226 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:15:53.064723 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:43393
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:53.147414 443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:53.147614 443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:53.147636 443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:53.148728 443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:53.148764 443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:53.148783 443 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:53.148799 443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:15:53.150226 443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:53.150417 443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:53.150600 443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:53.155470 451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:53.155474 454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:53.155471 452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:53.155611 443 server_base.cc:1034] running on GCE node
I20260810 15:15:53.155915 443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:53.156157 443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:53.157331 443 hybrid_clock.cc:648] HybridClock initialized: now 1786374953157315 us; error 35 us; skew 500 ppm
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:53.159857 443 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:15:53.160307 443 webserver.cc:460] Webserver started at http://127.29.230.2:40287/ using document root <none> and password file <none>
I20260810 15:15:53.160519 443 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:53.160563 443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:53.160692 443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:53.161751 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "713c797e0b7e4452888f963387099ac0"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "d3eba623c3edfe399759a39c15506a46"
server_key_iv: "1407438726d9c7e379d9f395f6a79030"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.162099 443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "713c797e0b7e4452888f963387099ac0"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "d3eba623c3edfe399759a39c15506a46"
server_key_iv: "1407438726d9c7e379d9f395f6a79030"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.163475 443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:15:53.164170 463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.164335 443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:53.164417 443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "713c797e0b7e4452888f963387099ac0"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "d3eba623c3edfe399759a39c15506a46"
server_key_iv: "1407438726d9c7e379d9f395f6a79030"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.164474 443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:53.192488 443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:53.194701 443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:53.194820 443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:53.195365 443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:53.195408 443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.195430 443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:53.195441 443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.202409 443 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:39917
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:53.202492 666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:39917 every 8 connection(s)
I20260810 15:15:53.202848 443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:15:53.205576 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.195788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:37967 (local address 127.29.230.62:43393)
0810 15:15:53.195851 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.195854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.196084 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:15:53.196768 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.196769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.196780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.196837 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.197741 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.198507 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.200056 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.200378 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.200444 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.203071 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.203072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.203072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.203080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.204483 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.204851 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.204852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.204852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.204901 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.205076 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.205077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.205077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.205180 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.205227 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.205450 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.205513 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:15:53.205763 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.195759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.2:37967)
0810 15:15:53.195937 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.195995 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.196634 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.196941 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.196946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.197334 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.197640 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.197649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.198612 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.198613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.199946 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.199952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.200478 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.200479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.200633 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.201181 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.201191 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.202938 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.204551 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.204557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.204559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.204789 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.204944 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.204944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.204944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.205033 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.205224 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.205228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.205344 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.205433 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.205555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260810 15:15:53.206269 667 heartbeater.cc:347] Connected to a master server at 127.29.230.62:43393
I20260810 15:15:53.206347 667 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:53.206554 667 heartbeater.cc:511] Master 127.29.230.62:43393 requested a full tablet report, sending...
I20260810 15:15:53.206998 32550 ts_manager.cc:194] Registered new tserver with Master: 713c797e0b7e4452888f963387099ac0 (127.29.230.2:39917)
I20260810 15:15:53.207515 32550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:37967
I20260810 15:15:53.210543 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 443
I20260810 15:15:53.210638 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:15:53.210806 30616 external_mini_cluster.cc:1428] Setting key f9c18c09e9c7d413bd7389b63f7a406c
I20260810 15:15:53.212699 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.207311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:38467 (local address 127.29.230.62:43393)
0810 15:15:53.207350 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.207351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.207426 (+ 75us) server_negotiation.cc:407] Connection header received
0810 15:15:53.207492 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.207493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.207501 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.207543 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.207889 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.208526 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.209246 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.209430 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.209502 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.210256 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.210258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.210258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.210265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.211634 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.211930 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.211932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.211932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.211986 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.212153 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.212154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.212154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.212333 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.212392 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.212533 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.212613 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:15:53.212713 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.207213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.2:38467)
0810 15:15:53.207370 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.207381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.207440 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.207549 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.207550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.207737 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.207812 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.207814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.208641 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.208642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.209167 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.209169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.209488 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.209488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.209542 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.209893 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.209897 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.210187 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.211711 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.211712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.211712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.211868 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.212037 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.212038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.212038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.212096 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.212395 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.212396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.212444 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.212502 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.212607 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:15:53.239310 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:43393
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:53.322723 683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:53.322960 683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:53.322991 683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:53.324275 683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:53.324328 683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:53.324357 683 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:53.324380 683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:15:53.325865 683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:43393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:53.326115 683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:53.326341 683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:53.331174 696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:53.331174 700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:53.331184 694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:53.331437 683 server_base.cc:1034] running on GCE node
I20260810 15:15:53.331643 683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:53.331840 683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:53.333014 683 hybrid_clock.cc:648] HybridClock initialized: now 1786374953332997 us; error 28 us; skew 500 ppm
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:53.335568 683 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:15:53.336055 683 webserver.cc:460] Webserver started at http://127.29.230.3:36703/ using document root <none> and password file <none>
I20260810 15:15:53.336266 683 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:53.336313 683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:53.336450 683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:53.337544 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "ef36ff096e034079981406de7fb08e19"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "b1ab9f9c95f12594330eb24c50118d0a"
server_key_iv: "7d28d86ec2c6bbfbf86bb803c9e3766b"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.337893 683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "ef36ff096e034079981406de7fb08e19"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "b1ab9f9c95f12594330eb24c50118d0a"
server_key_iv: "7d28d86ec2c6bbfbf86bb803c9e3766b"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.339184 683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:15:53.339988 715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.340209 683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:53.340325 683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "ef36ff096e034079981406de7fb08e19"
format_stamp: "Formatted at 2026-08-10 15:15:53 on dist-test-slave-gjrz"
server_key: "b1ab9f9c95f12594330eb24c50118d0a"
server_key_iv: "7d28d86ec2c6bbfbf86bb803c9e3766b"
server_key_version: "encryptionkey@0"
I20260810 15:15:53.340395 683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:53.351740 683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:53.353742 683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:53.353876 683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:53.354362 683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:53.354403 683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.354434 683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:53.354458 683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.360921 683 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:45649
I20260810 15:15:53.361083 918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:45649 every 8 connection(s)
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:53.361327 683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:15:53.363916 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.354859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:51269 (local address 127.29.230.62:43393)
0810 15:15:53.354901 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.354903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.355152 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:15:53.355742 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.355744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.355757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.355826 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.356582 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.357347 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.358798 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.359093 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.359161 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.361648 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.361649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.361650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.361658 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.362783 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.363133 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.363133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.363133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.363179 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.363387 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.363388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.363388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.363508 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.363560 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.363781 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.363852 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:15:53.364042 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.354867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.3:51269)
0810 15:15:53.355041 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.355077 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.355630 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.355884 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.355887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.356196 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.356476 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.356484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.357442 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.357443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.358722 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.358725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.359168 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.359169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.359337 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.359838 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.359848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.361535 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.362851 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.362856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.362857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.363074 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.363216 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.363217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.363217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.363342 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.363557 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.363561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.363675 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.363750 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.363871 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260810 15:15:53.364543 920 heartbeater.cc:347] Connected to a master server at 127.29.230.62:43393
I20260810 15:15:53.364624 920 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:53.364815 920 heartbeater.cc:511] Master 127.29.230.62:43393 requested a full tablet report, sending...
I20260810 15:15:53.364993 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 683
I20260810 15:15:53.365091 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:15:53.365164 32549 ts_manager.cc:194] Registered new tserver with Master: ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.365283 30616 external_mini_cluster.cc:1428] Setting key 9b81b5b6bfdb0fbe192498667a3ba720
I20260810 15:15:53.365645 32549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:51269
I20260810 15:15:53.366362 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:15:53.371606 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.365601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:43867 (local address 127.29.230.62:43393)
0810 15:15:53.365637 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.365638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.365840 (+ 202us) server_negotiation.cc:407] Connection header received
0810 15:15:53.365939 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.365940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.365951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.365995 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.366338 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.367018 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.367944 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.368137 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.368246 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.369142 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.369144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.369144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.369152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.370534 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.370849 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.370850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.370850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.370908 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.371073 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.371074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.371074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.371182 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.371239 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.371430 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.371510 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:15:53.371618 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.365544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.29.230.3:43867)
0810 15:15:53.365780 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.365791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.365866 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.366013 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.366014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.366216 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.366270 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.366272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.367173 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.367174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.367859 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.367861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.368249 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.368250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.368322 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.368770 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.368775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.369070 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.370612 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.370613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.370614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.370786 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.370961 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.370962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.370962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.371019 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.371238 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.371239 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.371335 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.371402 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.371511 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":71,"threads_started":1}
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:53.390324 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.383846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.29.230.62:43393)
0810 15:15:53.383954 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.383956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.383966 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:15:53.383993 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.383994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.384006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.384065 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.384576 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.385098 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.385837 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.386071 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.386132 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.388167 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.388168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.388168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.388175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.389413 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.389722 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.389723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.389723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.389780 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.389915 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.389916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.389916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.390042 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.390114 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.390190 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.390250 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:15:53.392010 32549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:15:53.393545 32549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:53.400128 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.395783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46247 (local address 127.0.0.1:51964)
0810 15:15:53.395976 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.396018 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.396085 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.396639 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.396640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.396642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:53.396698 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.396700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.398083 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.398085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.399128 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.399131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.399867 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.399940 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.399944 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.399996 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.400041 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260810 15:15:53.400215 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.396049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.29.230.3:45649)
0810 15:15:53.396243 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.396248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.396259 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:15:53.396301 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.396303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.396321 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.396408 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:53.396782 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.397523 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.398849 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.399627 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399693 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.399751 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.400084 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.400123 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260810 15:15:53.400363 945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.395783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:45649 (local address 127.0.0.1:49000)
0810 15:15:53.395976 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.396018 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.396085 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.396429 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.396433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.396435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:53.396508 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.396509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.397628 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.397629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.398753 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.398755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.399784 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.399881 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.399895 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.399957 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.400007 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260810 15:15:53.400141 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.396249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.29.230.1:46247)
0810 15:15:53.396434 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.396439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.396448 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:53.396482 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.396484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.396499 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.396579 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:53.396862 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.397988 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399217 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.399782 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399855 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.399926 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.400014 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.400052 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260810 15:15:53.401252 946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.395792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:39917 (local address 127.0.0.1:55356)
0810 15:15:53.396183 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.396190 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.396224 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.396735 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.396736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.396738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:53.396777 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.396779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.399217 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.399217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.400231 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.400234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.401055 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.401055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.401129 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.401133 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.401163 (+ 30us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.401200 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":94,"threads_started":1}
I20260810 15:15:53.401124 802 tablet_service.cc:1467] Processing CreateTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 (DEFAULT_TABLE table=test-table [id=eccf10d2a843499e94feb4b587ab4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.401122 321 tablet_service.cc:1467] Processing CreateTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 (DEFAULT_TABLE table=test-table [id=eccf10d2a843499e94feb4b587ab4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.401492 321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cab2a872cc433581f19d9c6a5f0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.401492 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cab2a872cc433581f19d9c6a5f0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.401625 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.396046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.29.230.2:39917)
0810 15:15:53.396479 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.396483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.396491 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:15:53.396526 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.396527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.396568 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.396676 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:53.396840 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.397995 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.400326 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.400966 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.401404 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.401470 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.401495 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.401533 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":84,"threads_started":1}
I20260810 15:15:53.402174 552 tablet_service.cc:1467] Processing CreateTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 (DEFAULT_TABLE table=test-table [id=eccf10d2a843499e94feb4b587ab4d87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.402508 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0cab2a872cc433581f19d9c6a5f0677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.405045 954 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Bootstrap starting.
I20260810 15:15:53.405305 956 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Bootstrap starting.
I20260810 15:15:53.405851 957 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Bootstrap starting.
I20260810 15:15:53.405997 956 tablet_bootstrap.cc:654] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.406087 954 tablet_bootstrap.cc:654] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.406394 954 log.cc:824] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:53.406392 956 log.cc:824] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:53.406509 957 tablet_bootstrap.cc:654] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.406813 957 log.cc:824] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:53.407186 954 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: No bootstrap required, opened a new log
I20260810 15:15:53.407312 954 ts_tablet_manager.cc:1397] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:53.407532 956 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: No bootstrap required, opened a new log
I20260810 15:15:53.407582 956 ts_tablet_manager.cc:1397] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:53.407673 957 tablet_bootstrap.cc:492] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: No bootstrap required, opened a new log
I20260810 15:15:53.407732 957 ts_tablet_manager.cc:1397] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:53.408862 956 raft_consensus.cc:348] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.408989 956 raft_consensus.cc:374] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.409010 956 raft_consensus.cc:729] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc0b38a5b8843f19719c49dfb3ec5e6, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.409103 956 consensus_queue.cc:260] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.409348 956 ts_tablet_manager.cc:1428] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:53.409370 428 heartbeater.cc:503] Master 127.29.230.62:43393 was elected leader, sending a full tablet report...
I20260810 15:15:53.409253 954 raft_consensus.cc:348] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.409402 954 raft_consensus.cc:374] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.409432 954 raft_consensus.cc:729] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef36ff096e034079981406de7fb08e19, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.409551 954 consensus_queue.cc:260] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.409560 957 raft_consensus.cc:348] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.409706 957 raft_consensus.cc:374] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.409739 957 raft_consensus.cc:729] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.409845 920 heartbeater.cc:503] Master 127.29.230.62:43393 was elected leader, sending a full tablet report...
I20260810 15:15:53.409845 957 consensus_queue.cc:260] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.409847 954 ts_tablet_manager.cc:1428] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 15:15:53.410109 957 ts_tablet_manager.cc:1428] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:15:53.410140 667 heartbeater.cc:503] Master 127.29.230.62:43393 was elected leader, sending a full tablet report...
I20260810 15:15:53.441236 968 raft_consensus.cc:482] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:53.441473 968 raft_consensus.cc:504] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.441829 968 leader_election.cc:288] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.444850 475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.442012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:45649 (local address 127.29.230.2:58251)
0810 15:15:53.442069 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.442081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.442126 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.442320 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.442321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.442323 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:53.442396 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.442398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.443089 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.443090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.443947 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.443950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.444552 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.444648 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.444652 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.444722 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.444763 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:15:53.444900 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.442135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:58251 (local address 127.29.230.3:45649)
0810 15:15:53.442193 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.442195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.442204 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:53.442233 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.442234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.442243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.442308 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:53.442466 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.443017 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444020 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.444460 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444517 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.444526 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.444772 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.444827 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:15:53.445134 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.442094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:49247 (local address 127.29.230.1:46247)
0810 15:15:53.442142 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.442144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.442296 (+ 152us) server_negotiation.cc:407] Connection header received
0810 15:15:53.442359 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.442360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.442370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.442423 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:53.442583 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.443132 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444261 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.444711 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444775 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.444783 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.445021 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.445072 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:15:53.445290 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cab2a872cc433581f19d9c6a5f0677" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19" is_pre_election: true
I20260810 15:15:53.445366 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.442037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46247 (local address 127.29.230.2:49247)
0810 15:15:53.442231 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.442242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.442302 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.442440 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.442441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.442443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:53.442523 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.442525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.443202 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.443202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.444153 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.444155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.444807 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.444813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.444895 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.444899 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.445225 (+ 326us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.445275 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":124,"threads_started":1}
I20260810 15:15:53.445482 836 raft_consensus.cc:2454] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.445708 466 leader_election.cc:302] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, ef36ff096e034079981406de7fb08e19; no voters:
I20260810 15:15:53.445732 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cab2a872cc433581f19d9c6a5f0677" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" is_pre_election: true
I20260810 15:15:53.445833 968 raft_consensus.cc:2781] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:53.445876 351 raft_consensus.cc:2454] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.445894 968 raft_consensus.cc:482] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:53.445925 968 raft_consensus.cc:3037] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.446995 968 raft_consensus.cc:504] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.447122 968 leader_election.cc:288] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Requested vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.447310 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cab2a872cc433581f19d9c6a5f0677" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
I20260810 15:15:53.447364 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0cab2a872cc433581f19d9c6a5f0677" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19"
I20260810 15:15:53.447392 351 raft_consensus.cc:3037] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.447453 836 raft_consensus.cc:3037] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.448124 836 raft_consensus.cc:2454] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.448125 351 raft_consensus.cc:2454] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.448292 468 leader_election.cc:302] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, edc0b38a5b8843f19719c49dfb3ec5e6; no voters:
I20260810 15:15:53.448397 968 raft_consensus.cc:2781] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:53.448570 968 raft_consensus.cc:686] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 1 LEADER]: Becoming Leader. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Running, Role: LEADER
I20260810 15:15:53.448660 968 consensus_queue.cc:237] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } }
I20260810 15:15:53.449600 32550 catalog_manager.cc:5495] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 713c797e0b7e4452888f963387099ac0 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } health_report { overall_health: HEALTHY } } }
W20260810 15:15:53.453490 668 tablet.cc:2369] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:53.459708 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.457319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.29.230.2:39917)
0810 15:15:53.457376 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.457378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.457386 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:15:53.457442 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.457443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.457461 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.457538 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:53.457660 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.458141 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.458900 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.459091 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.459145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.459377 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:53.459474 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:53.459550 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.459601 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.459624 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:15:53.461576 351 raft_consensus.cc:1261] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.461931 836 raft_consensus.cc:1261] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.461971 968 consensus_queue.cc:1035] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.462203 968 consensus_queue.cc:1035] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.464490 976 mvcc.cc:205] Tried to move back new op lower bound from 7316991809375440896 to 7316991809326362624. Current Snapshot: MvccSnapshot[applied={T|T < 7316991809375440896}]
I20260810 15:15:53.464856 980 mvcc.cc:205] Tried to move back new op lower bound from 7316991809375440896 to 7316991809326362624. Current Snapshot: MvccSnapshot[applied={T|T < 7316991809375440896}]
I20260810 15:15:53.465468 979 mvcc.cc:205] Tried to move back new op lower bound from 7316991809375440896 to 7316991809326362624. Current Snapshot: MvccSnapshot[applied={T|T < 7316991809375440896}]
I20260810 15:15:53.468853 32549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:53.469007 32549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:53.469800 32549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Sending DeleteTablet for 3 replicas of tablet f0cab2a872cc433581f19d9c6a5f0677
I20260810 15:15:53.470335 552 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:55356
I20260810 15:15:53.470333 802 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:49000
I20260810 15:15:53.470332 321 tablet_service.cc:1514] Processing DeleteTablet for tablet f0cab2a872cc433581f19d9c6a5f0677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:51964
I20260810 15:15:53.470651 998 tablet_replica.cc:330] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: stopping tablet replica
I20260810 15:15:53.470793 998 raft_consensus.cc:2229] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:53.470837 32549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 15:15:53.470919 998 raft_consensus.cc:2258] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 15:15:53.470984 32549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:53.471163 997 tablet_replica.cc:330] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: stopping tablet replica
I20260810 15:15:53.471294 997 raft_consensus.cc:2229] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:53.471421 997 raft_consensus.cc:2258] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:53.471671 998 ts_tablet_manager.cc:1905] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.471884 997 ts_tablet_manager.cc:1905] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.472393 321 tablet_service.cc:1467] Processing CreateTablet for tablet 718e1ed578d34978a791d538479e4564 (DEFAULT_TABLE table=test-table [id=38972885900445d8b44a9a27fd30d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.472435 802 tablet_service.cc:1467] Processing CreateTablet for tablet 718e1ed578d34978a791d538479e4564 (DEFAULT_TABLE table=test-table [id=38972885900445d8b44a9a27fd30d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.472522 321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e1ed578d34978a791d538479e4564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.472545 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e1ed578d34978a791d538479e4564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.473862 954 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Bootstrap starting.
I20260810 15:15:53.473908 956 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Bootstrap starting.
I20260810 15:15:53.473919 996 tablet_replica.cc:330] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: stopping tablet replica
I20260810 15:15:53.474009 996 raft_consensus.cc:2229] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:53.474138 996 raft_consensus.cc:2258] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:53.474169 997 ts_tablet_manager.cc:1918] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:15:53.474260 954 tablet_bootstrap.cc:654] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.474303 956 tablet_bootstrap.cc:654] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.474496 552 tablet_service.cc:1467] Processing CreateTablet for tablet 718e1ed578d34978a791d538479e4564 (DEFAULT_TABLE table=test-table [id=38972885900445d8b44a9a27fd30d2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:53.474584 996 ts_tablet_manager.cc:1905] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.474601 997 log.cc:1196] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/f0cab2a872cc433581f19d9c6a5f0677
I20260810 15:15:53.474613 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718e1ed578d34978a791d538479e4564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.474840 997 ts_tablet_manager.cc:1939] T f0cab2a872cc433581f19d9c6a5f0677 P ef36ff096e034079981406de7fb08e19: Deleting consensus metadata
I20260810 15:15:53.475217 32523 catalog_manager.cc:4841] TS ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649): tablet f0cab2a872cc433581f19d9c6a5f0677 (table test-table [id=eccf10d2a843499e94feb4b587ab4d87]) successfully deleted
I20260810 15:15:53.475400 998 ts_tablet_manager.cc:1918] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:15:53.475472 998 log.cc:1196] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/f0cab2a872cc433581f19d9c6a5f0677
I20260810 15:15:53.475718 998 ts_tablet_manager.cc:1939] T f0cab2a872cc433581f19d9c6a5f0677 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting consensus metadata
I20260810 15:15:53.475971 957 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Bootstrap starting.
I20260810 15:15:53.476101 32529 catalog_manager.cc:4841] TS edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247): tablet f0cab2a872cc433581f19d9c6a5f0677 (table test-table [id=eccf10d2a843499e94feb4b587ab4d87]) successfully deleted
I20260810 15:15:53.476300 996 ts_tablet_manager.cc:1918] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:15:53.476344 957 tablet_bootstrap.cc:654] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.476358 996 log.cc:1196] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/f0cab2a872cc433581f19d9c6a5f0677
I20260810 15:15:53.476598 996 ts_tablet_manager.cc:1939] T f0cab2a872cc433581f19d9c6a5f0677 P 713c797e0b7e4452888f963387099ac0: Deleting consensus metadata
I20260810 15:15:53.476926 32520 catalog_manager.cc:4841] TS 713c797e0b7e4452888f963387099ac0 (127.29.230.2:39917): tablet f0cab2a872cc433581f19d9c6a5f0677 (table test-table [id=eccf10d2a843499e94feb4b587ab4d87]) successfully deleted
I20260810 15:15:53.477010 957 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: No bootstrap required, opened a new log
I20260810 15:15:53.477073 957 ts_tablet_manager.cc:1397] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:53.477177 956 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: No bootstrap required, opened a new log
I20260810 15:15:53.477223 956 ts_tablet_manager.cc:1397] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 15:15:53.477228 957 raft_consensus.cc:348] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.477308 957 raft_consensus.cc:374] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.477332 957 raft_consensus.cc:729] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.477371 957 consensus_queue.cc:260] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.477358 956 raft_consensus.cc:348] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.477427 956 raft_consensus.cc:374] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.477452 956 raft_consensus.cc:729] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc0b38a5b8843f19719c49dfb3ec5e6, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.477458 957 ts_tablet_manager.cc:1428] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.477491 956 consensus_queue.cc:260] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.477576 956 ts_tablet_manager.cc:1428] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:53.479650 954 tablet_bootstrap.cc:492] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: No bootstrap required, opened a new log
I20260810 15:15:53.479696 954 ts_tablet_manager.cc:1397] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260810 15:15:53.479830 954 raft_consensus.cc:348] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.479893 954 raft_consensus.cc:374] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.479910 954 raft_consensus.cc:729] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef36ff096e034079981406de7fb08e19, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.479951 954 consensus_queue.cc:260] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.480041 954 ts_tablet_manager.cc:1428] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260810 15:15:53.533500 429 tablet.cc:2369] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:53.540808 985 raft_consensus.cc:482] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:53.540932 985 raft_consensus.cc:504] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.541157 985 leader_election.cc:288] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.541428 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e1ed578d34978a791d538479e4564" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" is_pre_election: true
I20260810 15:15:53.541428 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e1ed578d34978a791d538479e4564" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19" is_pre_election: true
I20260810 15:15:53.541538 836 raft_consensus.cc:2454] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.541538 351 raft_consensus.cc:2454] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.541700 468 leader_election.cc:302] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, edc0b38a5b8843f19719c49dfb3ec5e6; no voters:
I20260810 15:15:53.541798 985 raft_consensus.cc:2781] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:53.541832 985 raft_consensus.cc:482] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:53.541851 985 raft_consensus.cc:3037] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.542636 985 raft_consensus.cc:504] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.542832 985 leader_election.cc:288] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Requested vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.542913 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e1ed578d34978a791d538479e4564" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
I20260810 15:15:53.542997 351 raft_consensus.cc:3037] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.542976 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718e1ed578d34978a791d538479e4564" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19"
I20260810 15:15:53.543069 836 raft_consensus.cc:3037] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.543769 836 raft_consensus.cc:2454] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.543924 466 leader_election.cc:302] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, ef36ff096e034079981406de7fb08e19; no voters:
I20260810 15:15:53.543941 351 raft_consensus.cc:2454] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.544064 985 raft_consensus.cc:2781] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:53.544188 985 raft_consensus.cc:686] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 1 LEADER]: Becoming Leader. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Running, Role: LEADER
I20260810 15:15:53.544245 985 consensus_queue.cc:237] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.544855 32550 catalog_manager.cc:5495] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 713c797e0b7e4452888f963387099ac0 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } health_report { overall_health: UNKNOWN } } }
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374952, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:53.583217 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.576116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.29.230.62:43393)
0810 15:15:53.576167 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.576169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.576313 (+ 144us) server_negotiation.cc:407] Connection header received
0810 15:15:53.576383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.576384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.576393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.576459 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.576906 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.577391 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.578062 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.578239 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.578293 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.580695 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.580696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.580696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.580704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.582074 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.582423 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.582424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.582424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.582496 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.582691 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.582691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.582692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.582807 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.582869 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.583020 (+ 151us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.583112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
I20260810 15:15:53.583314 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.576079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:43393 (local address 127.0.0.1:33790)
0810 15:15:53.576239 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.576249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.576324 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.576469 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.576469 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.576779 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:53.576829 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.576832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.577467 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.577468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.577991 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.577993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.578297 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.578298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.578348 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.578912 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:53.578916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:53.580606 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:53.582154 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.582164 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.582166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.582357 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.582561 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:53.582561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:53.582562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:53.582622 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:53.582869 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:53.582879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:53.582927 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.582992 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.583180 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":77,"thread_start_us":92,"threads_started":1}
I20260810 15:15:53.585902 32550 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33790:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:15:53.586244 32550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:53.587659 321 tablet_service.cc:1467] Processing CreateTablet for tablet 28dcd025279f4fc0952b7e86ca365943 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd864e3a84924450b2cdaf1462e28295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:53.587694 802 tablet_service.cc:1467] Processing CreateTablet for tablet 28dcd025279f4fc0952b7e86ca365943 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd864e3a84924450b2cdaf1462e28295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:53.587858 321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dcd025279f4fc0952b7e86ca365943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.587860 802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dcd025279f4fc0952b7e86ca365943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.588939 956 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: Bootstrap starting.
I20260810 15:15:53.589265 956 tablet_bootstrap.cc:654] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.589359 954 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: Bootstrap starting.
I20260810 15:15:53.589723 954 tablet_bootstrap.cc:654] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.589859 956 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: No bootstrap required, opened a new log
I20260810 15:15:53.589906 956 ts_tablet_manager.cc:1397] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:53.590030 956 raft_consensus.cc:348] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.590087 956 raft_consensus.cc:374] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.590103 956 raft_consensus.cc:729] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc0b38a5b8843f19719c49dfb3ec5e6, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.587656 552 tablet_service.cc:1467] Processing CreateTablet for tablet 28dcd025279f4fc0952b7e86ca365943 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd864e3a84924450b2cdaf1462e28295]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:53.590147 956 consensus_queue.cc:260] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.590245 956 ts_tablet_manager.cc:1428] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.590267 552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28dcd025279f4fc0952b7e86ca365943. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:53.590389 966 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.590477 966 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.590548 954 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: No bootstrap required, opened a new log
I20260810 15:15:53.590587 954 ts_tablet_manager.cc:1397] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:53.590720 954 raft_consensus.cc:348] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.590812 954 raft_consensus.cc:374] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.590832 954 raft_consensus.cc:729] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef36ff096e034079981406de7fb08e19, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.590871 954 consensus_queue.cc:260] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.590961 954 ts_tablet_manager.cc:1428] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.591068 967 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.591174 967 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.591719 957 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: Bootstrap starting.
I20260810 15:15:53.592013 957 tablet_bootstrap.cc:654] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:53.592547 957 tablet_bootstrap.cc:492] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: No bootstrap required, opened a new log
I20260810 15:15:53.592604 957 ts_tablet_manager.cc:1397] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:53.592708 957 raft_consensus.cc:348] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.592761 957 raft_consensus.cc:374] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:53.592782 957 raft_consensus.cc:729] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Initialized, Role: FOLLOWER
I20260810 15:15:53.592831 957 consensus_queue.cc:260] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.592916 957 ts_tablet_manager.cc:1428] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:53.592944 985 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.593006 985 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 15:15:53.611959 922 tablet.cc:2369] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:53.713303 985 raft_consensus.cc:482] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:53.713406 985 raft_consensus.cc:504] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.713595 985 leader_election.cc:288] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.713845 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dcd025279f4fc0952b7e86ca365943" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" is_pre_election: true
I20260810 15:15:53.713856 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dcd025279f4fc0952b7e86ca365943" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19" is_pre_election: true
I20260810 15:15:53.713955 836 raft_consensus.cc:2454] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.713958 351 raft_consensus.cc:2454] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 0.
I20260810 15:15:53.714119 468 leader_election.cc:302] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, edc0b38a5b8843f19719c49dfb3ec5e6; no voters:
I20260810 15:15:53.714246 985 raft_consensus.cc:2781] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:53.714301 985 raft_consensus.cc:482] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:53.714322 985 raft_consensus.cc:3037] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.715018 985 raft_consensus.cc:504] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.715145 985 leader_election.cc:288] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Requested vote from peers edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247), ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649)
I20260810 15:15:53.715358 836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dcd025279f4fc0952b7e86ca365943" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef36ff096e034079981406de7fb08e19"
I20260810 15:15:53.715346 351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28dcd025279f4fc0952b7e86ca365943" candidate_uuid: "713c797e0b7e4452888f963387099ac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6"
I20260810 15:15:53.715433 351 raft_consensus.cc:3037] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.715431 836 raft_consensus.cc:3037] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:53.715989 836 raft_consensus.cc:2454] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.715987 351 raft_consensus.cc:2454] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 713c797e0b7e4452888f963387099ac0 in term 1.
I20260810 15:15:53.716171 468 leader_election.cc:302] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 713c797e0b7e4452888f963387099ac0, edc0b38a5b8843f19719c49dfb3ec5e6; no voters:
I20260810 15:15:53.716279 985 raft_consensus.cc:2781] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:53.716338 985 raft_consensus.cc:686] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [term 1 LEADER]: Becoming Leader. State: Replica: 713c797e0b7e4452888f963387099ac0, State: Running, Role: LEADER
I20260810 15:15:53.716389 985 consensus_queue.cc:237] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } }
I20260810 15:15:53.716681 971 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: TxnStatusTablet state changed. Reason: New leader 713c797e0b7e4452888f963387099ac0. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.716758 971 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:53.716907 1031 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:15:53.717026 1031 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:15:53.717078 32550 catalog_manager.cc:5495] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 713c797e0b7e4452888f963387099ac0 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } health_report { overall_health: UNKNOWN } } }
I20260810 15:15:53.739931 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.737728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.29.230.2:39917)
0810 15:15:53.737788 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.737790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.737810 (+ 20us) server_negotiation.cc:407] Connection header received
0810 15:15:53.737835 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.737836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.737848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.737889 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:53.738015 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.738484 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.739221 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.739459 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.739511 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.739677 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:53.739715 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:53.739759 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.739824 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.739849 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:15:53.739936 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.737573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:39917 (local address 127.0.0.1:55372)
0810 15:15:53.737685 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.737700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.737755 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.737904 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.737905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.737906 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:53.737959 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.737961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.738558 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.738559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.739142 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.739145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.739530 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.739531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.739590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.739606 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:53.739767 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:53.739774 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.739815 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.739868 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 15:15:53.741954 836 raft_consensus.cc:1261] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.742175 351 raft_consensus.cc:1261] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.742213 971 consensus_queue.cc:1035] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.742389 971 consensus_queue.cc:1035] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.742916 967 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: TxnStatusTablet state changed. Reason: New leader 713c797e0b7e4452888f963387099ac0. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.743026 967 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.743094 966 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: TxnStatusTablet state changed. Reason: New leader 713c797e0b7e4452888f963387099ac0. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.743180 966 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.743417 971 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.743506 971 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:53.744057 971 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.744134 971 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P 713c797e0b7e4452888f963387099ac0: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:53.744720 967 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.744800 967 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P ef36ff096e034079981406de7fb08e19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.745872 966 tablet_replica.cc:439] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "713c797e0b7e4452888f963387099ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 } } peers { permanent_uuid: "713c797e0b7e4452888f963387099ac0" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 39917 } } peers { permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 } } }
I20260810 15:15:53.745959 966 tablet_replica.cc:442] T 28dcd025279f4fc0952b7e86ca365943 P edc0b38a5b8843f19719c49dfb3ec5e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:53.750538 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.747910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:35609 (local address 127.29.230.2:39917)
0810 15:15:53.747957 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.747959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.748115 (+ 156us) server_negotiation.cc:407] Connection header received
0810 15:15:53.748168 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.748169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.748177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.748220 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:53.748343 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.748800 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.749882 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.750091 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.750145 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.750306 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:53.750332 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:53.750362 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.750418 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.750453 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":22}
I20260810 15:15:53.750680 1046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.747850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:39917 (local address 127.29.230.2:35609)
0810 15:15:53.748034 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:53.748047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:53.748113 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:53.748235 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:53.748235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:53.748237 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:53.748288 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.748290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.748856 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.748857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.749810 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:53.749813 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.750158 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.750159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:53.750215 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.750232 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:53.750375 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:53.750377 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:15:53.750417 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 15:15:53.750479 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":2,"thread_start_us":104,"threads_started":1}
I20260810 15:15:53.752653 351 raft_consensus.cc:1261] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.752657 836 raft_consensus.cc:1261] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Refusing update from remote peer 713c797e0b7e4452888f963387099ac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:53.752820 971 consensus_queue.cc:1035] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef36ff096e034079981406de7fb08e19" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 45649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.752920 968 consensus_queue.cc:1035] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edc0b38a5b8843f19719c49dfb3ec5e6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:53.764525 32548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:53.764649 32548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:53.765323 32548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c58ade0936f84f14890562383e2afd15: Sending DeleteTablet for 3 replicas of tablet 718e1ed578d34978a791d538479e4564
I20260810 15:15:53.765584 321 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e1ed578d34978a791d538479e4564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:51964
I20260810 15:15:53.765584 552 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e1ed578d34978a791d538479e4564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:55356
I20260810 15:15:53.765688 998 tablet_replica.cc:330] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: stopping tablet replica
I20260810 15:15:53.765699 996 tablet_replica.cc:330] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: stopping tablet replica
I20260810 15:15:53.765704 802 tablet_service.cc:1514] Processing DeleteTablet for tablet 718e1ed578d34978a791d538479e4564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:53 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:49000
I20260810 15:15:53.765734 998 raft_consensus.cc:2229] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:53.765758 996 raft_consensus.cc:2229] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:53.765769 998 raft_consensus.cc:2258] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:53.765781 997 tablet_replica.cc:330] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: stopping tablet replica
I20260810 15:15:53.765812 996 raft_consensus.cc:2258] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:53.765820 997 raft_consensus.cc:2229] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:53.765851 997 raft_consensus.cc:2258] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:53.766172 997 ts_tablet_manager.cc:1905] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.766762 998 ts_tablet_manager.cc:1905] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.767470 996 ts_tablet_manager.cc:1905] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:53.768002 997 ts_tablet_manager.cc:1918] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:53.768052 997 log.cc:1196] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/718e1ed578d34978a791d538479e4564
I20260810 15:15:53.768270 997 ts_tablet_manager.cc:1939] T 718e1ed578d34978a791d538479e4564 P ef36ff096e034079981406de7fb08e19: Deleting consensus metadata
I20260810 15:15:53.768325 998 ts_tablet_manager.cc:1918] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:53.768368 998 log.cc:1196] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/718e1ed578d34978a791d538479e4564
I20260810 15:15:53.768560 998 ts_tablet_manager.cc:1939] T 718e1ed578d34978a791d538479e4564 P edc0b38a5b8843f19719c49dfb3ec5e6: Deleting consensus metadata
I20260810 15:15:53.768680 32523 catalog_manager.cc:4841] TS ef36ff096e034079981406de7fb08e19 (127.29.230.3:45649): tablet 718e1ed578d34978a791d538479e4564 (table test-table [id=38972885900445d8b44a9a27fd30d2f1]) successfully deleted
I20260810 15:15:53.768939 32529 catalog_manager.cc:4841] TS edc0b38a5b8843f19719c49dfb3ec5e6 (127.29.230.1:46247): tablet 718e1ed578d34978a791d538479e4564 (table test-table [id=38972885900445d8b44a9a27fd30d2f1]) successfully deleted
I20260810 15:15:53.769164 996 ts_tablet_manager.cc:1918] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:53.769215 996 log.cc:1196] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/718e1ed578d34978a791d538479e4564
I20260810 15:15:53.769409 996 ts_tablet_manager.cc:1939] T 718e1ed578d34978a791d538479e4564 P 713c797e0b7e4452888f963387099ac0: Deleting consensus metadata
I20260810 15:15:53.769891 32520 catalog_manager.cc:4841] TS 713c797e0b7e4452888f963387099ac0 (127.29.230.2:39917): tablet 718e1ed578d34978a791d538479e4564 (table test-table [id=38972885900445d8b44a9a27fd30d2f1]) successfully deleted
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[32460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374953, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:15:53.778855 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.769997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.29.230.1:46247)
0810 15:15:53.770040 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.770042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.770247 (+ 205us) server_negotiation.cc:407] Connection header received
0810 15:15:53.770276 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.770277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.770288 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.770358 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.770954 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.771740 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.772483 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.772787 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.772857 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.775692 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.775706 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.775706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.775714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.777605 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.777932 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.777935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.777936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.778006 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.778230 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.778230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.778231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.778422 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.778584 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.778669 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.778756 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 15:15:53.779182 418 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51976
I20260810 15:15:53.786638 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:53.780257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.29.230.62:43393)
0810 15:15:53.780314 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:53.780317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:53.780366 (+ 49us) server_negotiation.cc:407] Connection header received
0810 15:15:53.780431 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:53.780432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:53.780450 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:53.780502 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:53.780983 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.781697 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.782400 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:53.782710 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:53.782791 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:53.784104 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:53.784106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:53.784106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:53.784114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:53.785635 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.785933 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.785934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.785935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.785995 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:53.786158 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:53.786158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:53.786159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:53.786285 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:53.786376 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:53.786466 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:15:53.786541 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 15:15:53.786906 32548 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33804
I20260810 15:15:53.787551 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32672
I20260810 15:15:53.808454 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 443
I20260810 15:15:53.818214 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 683
I20260810 15:15:53.836642 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32477
2026-08-10T15:15:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2443 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[1060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[1060](info): set up 2 sockets
Aug 10 15:15:53 dist-test-slave-gjrz krb5kdc[1060](info): commencing operation
krb5kdc: starting...
W20260810 15:15:55.039621 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:15:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:15:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:15:55.164459 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36653
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:42457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:55.247423 1088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:55.247653 1088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:55.247684 1088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:55.248840 1088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:15:55.248883 1088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:15:55.248899 1088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:55.248915 1088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:55.248930 1088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:15:55.248946 1088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:15:55.250464 1088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36653
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:55.250726 1088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:55.250941 1088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:55.255733 1097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.255721 1099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.255721 1103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:55.256273 1088 server_base.cc:1034] running on GCE node
I20260810 15:15:55.256526 1088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:55.256794 1088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:55.257961 1088 hybrid_clock.cc:648] HybridClock initialized: now 1786374955257946 us; error 29 us; skew 500 ppm
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:55.260564 1088 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:15:55.260993 1088 webserver.cc:460] Webserver started at http://127.29.230.62:41057/ using document root <none> and password file <none>
I20260810 15:15:55.261193 1088 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:55.261236 1088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:55.261356 1088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:55.262415 1088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "45086e201d56467a97e74130b8fa2f75"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "54b126fd6812686189af5aec785b266e"
server_key_iv: "35bfe79ef5317da4c6da2f4c402babc0"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.262765 1088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "45086e201d56467a97e74130b8fa2f75"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "54b126fd6812686189af5aec785b266e"
server_key_iv: "35bfe79ef5317da4c6da2f4c402babc0"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.264130 1088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:15:55.264835 1112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.265115 1088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:55.265237 1088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "45086e201d56467a97e74130b8fa2f75"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "54b126fd6812686189af5aec785b266e"
server_key_iv: "35bfe79ef5317da4c6da2f4c402babc0"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.265309 1088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:55.280177 1088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:55.282101 1088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:55.282243 1088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:55.287002 1088 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36653
I20260810 15:15:55.287098 1224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36653 every 8 connection(s)
I20260810 15:15:55.287443 1088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:15:55.288133 1225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.289805 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1088
I20260810 15:15:55.289880 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:15:55.290030 30616 external_mini_cluster.cc:1428] Setting key 7e9b0cd74238424ba38570c652710c44
I20260810 15:15:55.290783 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Bootstrap starting.
I20260810 15:15:55.291543 1225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.291877 1225 log.cc:824] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:55.292747 1225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: No bootstrap required, opened a new log
I20260810 15:15:55.295099 1225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } }
I20260810 15:15:55.295334 1225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.295441 1225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45086e201d56467a97e74130b8fa2f75, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.295589 1225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } }
I20260810 15:15:55.295672 1225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:15:55.295696 1225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:15:55.295743 1225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.296283 1225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } }
I20260810 15:15:55.296382 1225 leader_election.cc:302] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45086e201d56467a97e74130b8fa2f75; no voters:
I20260810 15:15:55.296556 1225 leader_election.cc:288] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:15:55.296615 1233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:55.296797 1225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:15:55.296808 1233 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [term 1 LEADER]: Becoming Leader. State: Replica: 45086e201d56467a97e74130b8fa2f75, State: Running, Role: LEADER
I20260810 15:15:55.296934 1233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } }
I20260810 15:15:55.297241 1233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45086e201d56467a97e74130b8fa2f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } } }
I20260810 15:15:55.297385 1233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [sys.catalog]: This master's current role is: LEADER
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.297580 1235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45086e201d56467a97e74130b8fa2f75. Latest consensus state: current_term: 1 leader_uuid: "45086e201d56467a97e74130b8fa2f75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45086e201d56467a97e74130b8fa2f75" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36653 } } }
I20260810 15:15:55.297688 1235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75 [sys.catalog]: This master's current role is: LEADER
W20260810 15:15:55.298837 1267 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:15:55.298938 1267 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:15:55.298998 1247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:15:55.299144 1247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:15:55.300765 1247 catalog_manager.cc:1347] Generated new cluster ID: 1fa8d0574adb4f1c9495bb602ed8d1a2
I20260810 15:15:55.300822 1247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:15:55.301386 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.290592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.29.230.62:36653)
0810 15:15:55.290779 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.290789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.290823 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:15:55.291459 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.291469 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.291536 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.291688 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.292389 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.293882 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.294667 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.295322 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.295438 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.297990 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.297996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.297996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.298008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.299746 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.300136 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.300139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.300139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.300254 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.300488 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.300489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.300490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.300773 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.300941 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.301095 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.301231 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260810 15:15:55.323005 1247 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:15:55.323796 1247 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:15:55.335587 1247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Generated new TSK 0
I20260810 15:15:55.335819 1247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:15:55.373915 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36653
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:42457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:55.456640 1274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:55.456844 1274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:55.456869 1274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:55.458011 1274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:55.458057 1274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:55.458077 1274 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:15:55.458088 1274 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:55.458118 1274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:15:55.459643 1274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:55.459872 1274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:55.460093 1274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:55.465130 1285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.465168 1295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.465268 1289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:55.465337 1274 server_base.cc:1034] running on GCE node
I20260810 15:15:55.465608 1274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:55.465837 1274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:55.466984 1274 hybrid_clock.cc:648] HybridClock initialized: now 1786374955466969 us; error 34 us; skew 500 ppm
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:55.469595 1274 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:15:55.470029 1274 webserver.cc:460] Webserver started at http://127.29.230.1:43201/ using document root <none> and password file <none>
I20260810 15:15:55.470232 1274 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:55.470276 1274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:55.470398 1274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:55.471488 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "6b1eff94ba022a2cecd8ec6b7a63d28a"
server_key_iv: "b4cfe3781e0267c0dacd6115d4535c9b"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.471825 1274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "6b1eff94ba022a2cecd8ec6b7a63d28a"
server_key_iv: "b4cfe3781e0267c0dacd6115d4535c9b"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.473207 1274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:15:55.473956 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.474164 1274 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:55.474251 1274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "6b1eff94ba022a2cecd8ec6b7a63d28a"
server_key_iv: "b4cfe3781e0267c0dacd6115d4535c9b"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.474321 1274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:55.486858 1274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:55.489022 1274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:55.489153 1274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:55.489729 1274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:55.489776 1274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.489800 1274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:55.489813 1274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.497164 1274 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:46073
I20260810 15:15:55.497669 1274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:15:55.498497 1529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:46073 every 8 connection(s)
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.500190 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1274
I20260810 15:15:55.500283 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:15:55.500463 30616 external_mini_cluster.cc:1428] Setting key 4134d5be90280006c6f2c6415049f8a0
I20260810 15:15:55.503378 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.490276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:35579 (local address 127.29.230.62:36653)
0810 15:15:55.490343 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.490345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.490548 (+ 203us) server_negotiation.cc:407] Connection header received
0810 15:15:55.491210 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.491212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.491226 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.491329 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.492352 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.493224 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.494842 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.495181 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.495258 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.500232 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.500235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.500235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.500246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.501762 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.502273 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.502274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.502275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.502337 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.502603 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.502603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.502604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.502715 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.502769 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.503117 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.503243 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:15:55.503485 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.490212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.1:35579)
0810 15:15:55.490391 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.490458 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.491067 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.491492 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.491498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.491880 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.492221 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.492229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.493338 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.493339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.494730 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.494734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.495334 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.495335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.495500 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.496352 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.496363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.500032 (+ 3669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.501850 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.501859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.501862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.502192 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.502399 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.502400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.502401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.502547 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.502771 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.502776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.502954 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.503077 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.503239 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260810 15:15:55.504251 1531 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36653
I20260810 15:15:55.504345 1531 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:55.504549 1531 heartbeater.cc:511] Master 127.29.230.62:36653 requested a full tablet report, sending...
I20260810 15:15:55.505125 1138 ts_manager.cc:194] Registered new tserver with Master: 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073)
I20260810 15:15:55.506021 1138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:35579
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:15:55.510923 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.505432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.1:34039)
0810 15:15:55.505574 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.505585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.505699 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.505869 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.505870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.506071 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.506142 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.506145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.506814 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.506815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.507372 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.507374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.507710 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.507711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.507788 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.508228 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.508231 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.508529 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.509964 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.509965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.509966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.510144 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.510403 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.510404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.510404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.510461 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.510667 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.510668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.510705 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.510752 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.510832 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260810 15:15:55.510984 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.505508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:34039 (local address 127.29.230.62:36653)
0810 15:15:55.505549 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.505551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.505671 (+ 120us) server_negotiation.cc:407] Connection header received
0810 15:15:55.505761 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.505761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.505771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.505797 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.506236 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.506728 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.507440 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.507648 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.507707 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.508614 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.508616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.508616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.508623 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.509889 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.510261 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.510263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.510263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.510331 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.510508 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.510508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.510509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.510622 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.510673 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.510819 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.510913 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:15:55.529747 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36653
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:42457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:55.612650 1546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:55.612875 1546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:55.612900 1546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:55.614018 1546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:55.614058 1546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:55.614076 1546 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:15:55.614086 1546 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:55.614101 1546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:15:55.615618 1546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:55.615831 1546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:55.616024 1546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:55.620774 1560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.620781 1554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.620834 1558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:55.621090 1546 server_base.cc:1034] running on GCE node
I20260810 15:15:55.621287 1546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:55.621506 1546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:55.622658 1546 hybrid_clock.cc:648] HybridClock initialized: now 1786374955622639 us; error 28 us; skew 500 ppm
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:55.625221 1546 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:15:55.625653 1546 webserver.cc:460] Webserver started at http://127.29.230.2:44815/ using document root <none> and password file <none>
I20260810 15:15:55.625867 1546 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:55.625911 1546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:55.626049 1546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:55.627063 1546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "3e6b00349cc842efb1c563ab05973f64"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "133092e6b3b993fc34e9d2d7bf10fa3b"
server_key_iv: "cec66d38c58209652c50aaf102d57a45"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.627418 1546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "3e6b00349cc842efb1c563ab05973f64"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "133092e6b3b993fc34e9d2d7bf10fa3b"
server_key_iv: "cec66d38c58209652c50aaf102d57a45"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.628669 1546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:55.629395 1573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.629542 1546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:15:55.629621 1546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "3e6b00349cc842efb1c563ab05973f64"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "133092e6b3b993fc34e9d2d7bf10fa3b"
server_key_iv: "cec66d38c58209652c50aaf102d57a45"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.629688 1546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:55.643992 1546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:55.645881 1546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:55.646049 1546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:55.646602 1546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:55.646646 1546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.646672 1546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:55.646684 1546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.654081 1546 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:41191
I20260810 15:15:55.654115 1801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:41191 every 8 connection(s)
I20260810 15:15:55.654503 1546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:15:55.655089 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1546
I20260810 15:15:55.655187 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:15:55.655427 30616 external_mini_cluster.cc:1428] Setting key 391ab8cc9993b9d61ec3f8fd953ad011
I20260810 15:15:55.656517 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.647058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:44497 (local address 127.29.230.62:36653)
0810 15:15:55.647161 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.647163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.647376 (+ 213us) server_negotiation.cc:407] Connection header received
0810 15:15:55.648069 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.648071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.648082 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.648124 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.649005 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.649604 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.650608 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.650915 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.650985 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.653387 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.653390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.653390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.653400 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.655030 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.655509 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.655511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.655511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.655578 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.655826 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.655827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.655827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.655996 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.656057 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.656337 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.656422 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 15:15:55.656682 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.647028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.2:44497)
0810 15:15:55.647214 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.647296 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.647940 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.648227 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.648232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.648602 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.648905 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.648912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.649697 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.649698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.650514 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.650516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.651003 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.651009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.651147 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.651644 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.651655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.653300 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.655117 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.655122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.655124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.655434 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.655643 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.655644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.655645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.655755 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.656059 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.656066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.656205 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.656288 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.656472 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260810 15:15:55.657943 1802 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36653
I20260810 15:15:55.658034 1802 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:55.658212 1802 heartbeater.cc:511] Master 127.29.230.62:36653 requested a full tablet report, sending...
I20260810 15:15:55.658563 1139 ts_manager.cc:194] Registered new tserver with Master: 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:55.659030 1139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:44497
I20260810 15:15:55.664767 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.658098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.2:58833)
0810 15:15:55.658421 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.658432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.659145 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.659345 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.659346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.659552 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.659611 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.659613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.660304 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.660305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.661037 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.661040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.661432 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.661433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.661511 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.661965 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.661970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.662314 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.663766 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.663767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.663768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.663927 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.664094 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.664095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.664095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.664156 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.664447 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.664449 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.664496 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.664557 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.664667 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":65,"threads_started":1}
I20260810 15:15:55.664762 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.658097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:58833 (local address 127.29.230.62:36653)
0810 15:15:55.658180 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.658183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.658478 (+ 295us) server_negotiation.cc:407] Connection header received
0810 15:15:55.659207 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.659208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.659220 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.659336 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.659707 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.660203 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.661158 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.661347 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.661413 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.662397 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.662399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.662399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.662406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.663693 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.663987 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.663988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.663988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.664041 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.664210 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.664211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.664211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.664386 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.664447 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.664586 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.664666 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:15:55.682320 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36653
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:42457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:15:55.764673 1818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:15:55.764907 1818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:15:55.764942 1818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:15:55.766099 1818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:15:55.766142 1818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:15:55.766170 1818 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 15:15:55.766184 1818 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:15:55.766208 1818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:15:55.767786 1818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:36653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:15:55.768046 1818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:15:55.768314 1818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:15:55.773229 1827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.773236 1826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:15:55.773283 1832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:15:55.773442 1818 server_base.cc:1034] running on GCE node
I20260810 15:15:55.773665 1818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:15:55.773871 1818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:15:55.775008 1818 hybrid_clock.cc:648] HybridClock initialized: now 1786374955774995 us; error 29 us; skew 500 ppm
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:15:55.780656 1818 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:15:55.781101 1818 webserver.cc:460] Webserver started at http://127.29.230.3:36785/ using document root <none> and password file <none>
I20260810 15:15:55.781327 1818 fs_manager.cc:359] Metadata directory not provided
I20260810 15:15:55.781370 1818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:15:55.781478 1818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:15:55.782533 1818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "b60d32461f56494fad52ea8e5a87087b"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "d4dae8bd8db94d7ea2c64df910381f74"
server_key_iv: "d228a3207e253069db69575dec3425c8"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.782867 1818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "b60d32461f56494fad52ea8e5a87087b"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "d4dae8bd8db94d7ea2c64df910381f74"
server_key_iv: "d228a3207e253069db69575dec3425c8"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.784245 1818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:15:55.784979 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.785154 1818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:15:55.785250 1818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "b60d32461f56494fad52ea8e5a87087b"
format_stamp: "Formatted at 2026-08-10 15:15:55 on dist-test-slave-gjrz"
server_key: "d4dae8bd8db94d7ea2c64df910381f74"
server_key_iv: "d228a3207e253069db69575dec3425c8"
server_key_version: "encryptionkey@0"
I20260810 15:15:55.785312 1818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:15:55.801463 1818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:15:55.803442 1818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:15:55.803583 1818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:15:55.804066 1818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:15:55.804114 1818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.804136 1818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:15:55.804149 1818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.811012 1818 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:34123
I20260810 15:15:55.811089 2043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:34123 every 8 connection(s)
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.811430 1818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:15:55.814139 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.804547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:33169 (local address 127.29.230.62:36653)
0810 15:15:55.804601 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.804604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.804836 (+ 232us) server_negotiation.cc:407] Connection header received
0810 15:15:55.805575 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.805577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.805590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.805644 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.806434 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.807203 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.808781 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.809142 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.809204 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.811893 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.811895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.811895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.811904 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.812954 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.813347 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.813348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.813348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.813412 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.813634 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.813635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.813635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.813747 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.813792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.814002 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.814076 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:15:55.814306 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.804551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.3:33169)
0810 15:15:55.804724 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.804763 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.805443 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.805719 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.805724 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.806071 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.806348 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.806354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.807331 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.807333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.808664 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.808668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.809228 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.809229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.809393 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.810007 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.810019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.811806 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.813029 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.813035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.813037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.813259 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.813481 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.813481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.813481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.813573 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.813790 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.813793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.813912 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.813985 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.814111 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260810 15:15:55.814805 2044 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36653
I20260810 15:15:55.814888 2044 heartbeater.cc:464] Registering TS with master...
I20260810 15:15:55.815068 2044 heartbeater.cc:511] Master 127.29.230.62:36653 requested a full tablet report, sending...
I20260810 15:15:55.815460 1138 ts_manager.cc:194] Registered new tserver with Master: b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123)
I20260810 15:15:55.816052 1138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:33169
I20260810 15:15:55.817906 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1818
I20260810 15:15:55.818022 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:15:55.818184 30616 external_mini_cluster.cc:1428] Setting key fef0c297a793675488ec67d33a12355e
I20260810 15:15:55.819015 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:15:55.821491 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.815994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:50227 (local address 127.29.230.62:36653)
0810 15:15:55.816038 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.816040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.816084 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:15:55.816167 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.816168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.816178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.816225 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.816609 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.817186 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.817913 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.818152 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.818204 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.819198 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.819199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.819200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.819206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.820426 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.820775 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.820776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.820776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.820821 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.820986 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.820986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.820986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.821077 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.821140 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.821299 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.821397 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:15:55.821503 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.815872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.29.230.3:50227)
0810 15:15:55.816024 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.816034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.816092 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.816239 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.816240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.816440 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.816507 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.816509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.817291 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.817292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.817831 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.817834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.818219 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.818220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.818294 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.818750 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.818755 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.819088 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.820508 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.820510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.820510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.820710 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.820875 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.820876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.820876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.820938 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.821156 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.821157 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.821207 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.821268 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.821399 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.843143 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.836395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.29.230.62:36653)
0810 15:15:55.836445 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.836447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.836456 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:55.836484 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.836484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.836494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.836548 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.837163 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.837677 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.838481 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.838677 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.838731 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.840909 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.840911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.840911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.840920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.842125 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.842435 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.842437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.842437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.842489 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.842665 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.842666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.842666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.842772 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.842860 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.842973 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.843048 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:15:55.844695 1138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:15:55.846361 1138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:55.852871 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.0.0.1:40348)
0810 15:15:55.849074 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.849082 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.849151 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.849642 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.849643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.849645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.849695 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.849696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850521 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.850522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.851899 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.851903 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852663 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.852715 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852719 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.852760 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.852798 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":64,"thread_start_us":68,"threads_started":1}
I20260810 15:15:55.852871 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46073 (local address 127.0.0.1:33606)
0810 15:15:55.848968 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.848976 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.849011 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.849375 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.849376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.849379 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.849435 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.849437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850681 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.850682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.851655 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.851658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852555 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.852639 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852653 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.852723 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.852777 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":79,"threads_started":1}
I20260810 15:15:55.853045 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.29.230.1:46073)
0810 15:15:55.849177 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.849181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.849191 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:15:55.849226 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.849228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.849240 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.849302 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.849569 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850608 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.851729 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852451 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852536 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852599 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.852882 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.852927 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":94,"threads_started":1}
I20260810 15:15:55.853118 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.29.230.3:34123)
0810 15:15:55.849418 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.849422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.849431 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:55.849472 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.849474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.849489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.849570 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.849762 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850435 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852000 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852504 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852570 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852630 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.852992 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.853032 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":131,"threads_started":1}
I20260810 15:15:55.853739 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:41191 (local address 127.0.0.1:48758)
0810 15:15:55.848728 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.848767 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.848831 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.849052 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.849056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.849058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.849133 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.849134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850504 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.850505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.851900 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.851903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852833 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.852902 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852906 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.853621 (+ 715us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.853670 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":125,"thread_start_us":69,"threads_started":1}
I20260810 15:15:55.853729 1379 tablet_service.cc:1467] Processing CreateTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 (DEFAULT_TABLE table=test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:55.853783 1941 tablet_service.cc:1467] Processing CreateTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 (DEFAULT_TABLE table=test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:55.853974 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.848681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48758 (local address 127.29.230.2:41191)
0810 15:15:55.848877 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.848882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.848893 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:15:55.848929 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.848931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.848947 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.849041 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.849237 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.850414 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.851985 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.852684 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.852744 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.852797 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.853840 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.853885 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
I20260810 15:15:55.854115 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e0e27eda46404d9ddca27fc5562cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.854146 1941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e0e27eda46404d9ddca27fc5562cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.856617 2087 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Bootstrap starting.
I20260810 15:15:55.857293 2089 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Bootstrap starting.
I20260810 15:15:55.857272 1673 tablet_service.cc:1467] Processing CreateTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 (DEFAULT_TABLE table=test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:55.857559 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e0e27eda46404d9ddca27fc5562cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.857764 2087 tablet_bootstrap.cc:654] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.857905 2089 tablet_bootstrap.cc:654] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.858165 2087 log.cc:824] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:55.858242 2089 log.cc:824] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:55.858914 2087 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: No bootstrap required, opened a new log
I20260810 15:15:55.858906 2089 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: No bootstrap required, opened a new log
I20260810 15:15:55.858979 2089 ts_tablet_manager.cc:1397] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:55.858979 2087 ts_tablet_manager.cc:1397] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:55.860860 2099 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Bootstrap starting.
I20260810 15:15:55.860924 2089 raft_consensus.cc:348] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.861099 2089 raft_consensus.cc:374] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.861145 2089 raft_consensus.cc:729] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.861164 2087 raft_consensus.cc:348] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.861254 2089 consensus_queue.cc:260] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.861305 2087 raft_consensus.cc:374] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.861343 2087 raft_consensus.cc:729] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.861466 2087 consensus_queue.cc:260] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.861508 2099 tablet_bootstrap.cc:654] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.861519 2089 ts_tablet_manager.cc:1428] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:15:55.861685 1531 heartbeater.cc:503] Master 127.29.230.62:36653 was elected leader, sending a full tablet report...
I20260810 15:15:55.861716 2087 ts_tablet_manager.cc:1428] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:15:55.861853 2099 log.cc:824] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Log is configured to *not* fsync() on all Append() calls
I20260810 15:15:55.861873 2044 heartbeater.cc:503] Master 127.29.230.62:36653 was elected leader, sending a full tablet report...
I20260810 15:15:55.862582 2099 tablet_bootstrap.cc:492] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: No bootstrap required, opened a new log
I20260810 15:15:55.862658 2099 ts_tablet_manager.cc:1397] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:55.864225 2099 raft_consensus.cc:348] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.864348 2099 raft_consensus.cc:374] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.864368 2099 raft_consensus.cc:729] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.864451 2099 consensus_queue.cc:260] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.864678 2099 ts_tablet_manager.cc:1428] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:15:55.864741 1802 heartbeater.cc:503] Master 127.29.230.62:36653 was elected leader, sending a full tablet report...
W20260810 15:15:55.905102 1804 tablet.cc:2369] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:55.913625 2101 raft_consensus.cc:482] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:55.913762 2101 raft_consensus.cc:504] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.914153 2101 leader_election.cc:288] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191), b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123)
I20260810 15:15:55.917243 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.914445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:43861 (local address 127.29.230.3:34123)
0810 15:15:55.914533 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.914535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.914549 (+ 14us) server_negotiation.cc:407] Connection header received
0810 15:15:55.914596 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.914596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.914606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.914684 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.914919 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.915605 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.916567 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.916926 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.916988 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.916997 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.917145 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.917179 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 15:15:55.917258 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.914457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53227 (local address 127.29.230.2:41191)
0810 15:15:55.914532 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.914534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.914727 (+ 193us) server_negotiation.cc:407] Connection header received
0810 15:15:55.914759 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.914761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.914772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.914844 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.914992 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.915590 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.916577 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.916945 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.917008 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.917017 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.917157 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.917190 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:15:55.917269 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.914348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.29.230.1:43861)
0810 15:15:55.914405 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.914416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.914457 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.914741 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.914742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.914745 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.914825 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.914827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.915677 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.915678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.916471 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.916474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.916999 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.917000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.917079 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.917083 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.917127 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.917164 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":5}
I20260810 15:15:55.917277 2108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.914370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:41191 (local address 127.29.230.1:53227)
0810 15:15:55.914624 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.914633 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.914686 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.914851 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.914851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.914853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.914924 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.914925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.915691 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.915692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.916480 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.916483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.917019 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.917021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.917093 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.917096 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.917140 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.917180 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":191,"threads_started":1}
I20260810 15:15:55.917603 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e0e27eda46404d9ddca27fc5562cd7" candidate_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60d32461f56494fad52ea8e5a87087b" is_pre_election: true
I20260810 15:15:55.917768 1970 raft_consensus.cc:2454] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b1e15b9c81440fd9903ea7a11f7c1bc in term 0.
I20260810 15:15:55.917753 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e0e27eda46404d9ddca27fc5562cd7" candidate_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64" is_pre_election: true
I20260810 15:15:55.917874 1707 raft_consensus.cc:2454] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b1e15b9c81440fd9903ea7a11f7c1bc in term 0.
I20260810 15:15:55.918016 1317 leader_election.cc:302] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:55.918171 2101 raft_consensus.cc:2781] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:55.918218 2101 raft_consensus.cc:482] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:55.918233 2101 raft_consensus.cc:3037] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.918897 2101 raft_consensus.cc:504] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.919015 2101 leader_election.cc:288] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [CANDIDATE]: Term 1 election: Requested vote from peers 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191), b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123)
I20260810 15:15:55.919171 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e0e27eda46404d9ddca27fc5562cd7" candidate_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60d32461f56494fad52ea8e5a87087b"
I20260810 15:15:55.919180 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19e0e27eda46404d9ddca27fc5562cd7" candidate_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64"
I20260810 15:15:55.919257 1970 raft_consensus.cc:3037] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.919283 1707 raft_consensus.cc:3037] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.919952 1707 raft_consensus.cc:2454] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b1e15b9c81440fd9903ea7a11f7c1bc in term 1.
I20260810 15:15:55.920012 1970 raft_consensus.cc:2454] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b1e15b9c81440fd9903ea7a11f7c1bc in term 1.
I20260810 15:15:55.920475 1314 leader_election.cc:302] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, 3e6b00349cc842efb1c563ab05973f64; no voters:
I20260810 15:15:55.920576 2101 raft_consensus.cc:2781] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:55.920722 2101 raft_consensus.cc:686] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 LEADER]: Becoming Leader. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Running, Role: LEADER
I20260810 15:15:55.920807 2101 consensus_queue.cc:237] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.921643 1139 catalog_manager.cc:5495] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1). New cstate: current_term: 1 leader_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } health_report { overall_health: UNKNOWN } } }
Aug 10 15:15:55 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374955, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:55.946732 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.939987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36653 (local address 127.0.0.1:60264)
0810 15:15:55.940177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.940188 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.940255 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.940371 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.940372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.940692 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:15:55.940744 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.940746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.941392 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.941393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.941935 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.941937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.942278 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.942278 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.942317 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.942771 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:15:55.942775 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:15:55.944156 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:15:55.945675 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.945683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.945686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.945886 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.946118 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:15:55.946119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:15:55.946119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:15:55.946187 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:15:55.946439 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:15:55.946451 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:15:55.946500 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.946555 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.946655 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260810 15:15:55.946825 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.940066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60264 (local address 127.29.230.62:36653)
0810 15:15:55.940157 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.940160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.940247 (+ 87us) server_negotiation.cc:407] Connection header received
0810 15:15:55.940309 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.940310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.940318 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.940357 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:55.940813 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.941334 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.942014 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.942219 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.942276 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.944259 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:55.944260 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:55.944261 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:55.944268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:55.945580 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.945974 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.945975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.945975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.946035 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:55.946265 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:55.946265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:55.946266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:55.946377 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:55.946435 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.946617 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.946713 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":63}
I20260810 15:15:55.948678 1139 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:15:55.948884 1139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:55.950416 1379 tablet_service.cc:1467] Processing CreateTablet for tablet 28444dadb82044a0b2ea64287f7f5337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=873a5b3734274b8bad52ca6090f1cfac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:55.950461 1941 tablet_service.cc:1467] Processing CreateTablet for tablet 28444dadb82044a0b2ea64287f7f5337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=873a5b3734274b8bad52ca6090f1cfac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:55.950480 1673 tablet_service.cc:1467] Processing CreateTablet for tablet 28444dadb82044a0b2ea64287f7f5337 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=873a5b3734274b8bad52ca6090f1cfac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:15:55.950608 1941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28444dadb82044a0b2ea64287f7f5337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.950605 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28444dadb82044a0b2ea64287f7f5337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.950639 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28444dadb82044a0b2ea64287f7f5337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:55.951828 2087 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: Bootstrap starting.
I20260810 15:15:55.952220 2089 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Bootstrap starting.
I20260810 15:15:55.952442 2087 tablet_bootstrap.cc:654] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.952631 2089 tablet_bootstrap.cc:654] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.952826 2099 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: Bootstrap starting.
I20260810 15:15:55.953133 2087 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: No bootstrap required, opened a new log
I20260810 15:15:55.953181 2087 ts_tablet_manager.cc:1397] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:55.953328 2087 raft_consensus.cc:348] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.953408 2087 raft_consensus.cc:374] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.953404 2089 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: No bootstrap required, opened a new log
I20260810 15:15:55.953429 2087 raft_consensus.cc:729] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.953442 2089 ts_tablet_manager.cc:1397] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:55.953483 2087 consensus_queue.cc:260] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.953503 2099 tablet_bootstrap.cc:654] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:55.953583 2087 ts_tablet_manager.cc:1428] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:55.953586 2089 raft_consensus.cc:348] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.953665 2089 raft_consensus.cc:374] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.953691 2089 raft_consensus.cc:729] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.953734 2089 consensus_queue.cc:260] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.953747 2102 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:55.953816 2089 ts_tablet_manager.cc:1428] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.953819 2102 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:55.953914 2101 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:55.954013 2101 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:55.954517 2099 tablet_bootstrap.cc:492] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: No bootstrap required, opened a new log
I20260810 15:15:55.954568 2099 ts_tablet_manager.cc:1397] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:55.954697 2099 raft_consensus.cc:348] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.954759 2099 raft_consensus.cc:374] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:55.954777 2099 raft_consensus.cc:729] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Initialized, Role: FOLLOWER
I20260810 15:15:55.954816 2099 consensus_queue.cc:260] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.954900 2099 ts_tablet_manager.cc:1428] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:55.954988 2107 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:55.955084 2107 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:55.992116 2102 raft_consensus.cc:482] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:55.992228 2102 raft_consensus.cc:504] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.992683 2102 leader_election.cc:288] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:55.996515 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.992900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:49769 (local address 127.29.230.1:46073)
0810 15:15:55.992947 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.992949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.992969 (+ 20us) server_negotiation.cc:407] Connection header received
0810 15:15:55.993107 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.993108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.993118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.993176 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.993348 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.994102 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.995576 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.996137 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.996205 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.996212 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.996396 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.996444 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:15:55.996511 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.992917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:48137 (local address 127.29.230.2:41191)
0810 15:15:55.992958 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:55.992960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:55.993142 (+ 182us) server_negotiation.cc:407] Connection header received
0810 15:15:55.993170 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:55.993171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:55.993179 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:55.993230 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:55.993348 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.994095 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.995588 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.996130 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.996199 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.996208 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:55.996395 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 15:15:55.996440 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:15:55.996546 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.992866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46073 (local address 127.29.230.3:49769)
0810 15:15:55.992922 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.992933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.992977 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.993197 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.993198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.993200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.993285 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.993287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.994184 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.994185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.995504 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.995505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.996222 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.996224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.996316 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.996320 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.996378 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.996447 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":16}
I20260810 15:15:55.996549 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:55.992866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:41191 (local address 127.29.230.3:48137)
0810 15:15:55.993069 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:55.993079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:55.993133 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:55.993235 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:55.993236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:55.993237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:55.993286 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.993287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.994184 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.994185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.995490 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:55.995494 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:55.996244 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:55.996245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:55.996316 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:55.996320 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:55.996378 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:15:55.996433 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":104,"thread_start_us":142,"threads_started":1}
I20260810 15:15:55.996892 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28444dadb82044a0b2ea64287f7f5337" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" is_pre_election: true
I20260810 15:15:55.997022 1413 raft_consensus.cc:2454] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:55.997038 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28444dadb82044a0b2ea64287f7f5337" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64" is_pre_election: true
I20260810 15:15:55.997143 1707 raft_consensus.cc:2454] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:55.997260 1852 leader_election.cc:302] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:55.997416 2102 raft_consensus.cc:2781] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:55.997466 2102 raft_consensus.cc:482] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:55.997488 2102 raft_consensus.cc:3037] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.998154 2102 raft_consensus.cc:504] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:55.998286 2102 leader_election.cc:288] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Requested vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:55.998469 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28444dadb82044a0b2ea64287f7f5337" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64"
I20260810 15:15:55.998477 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28444dadb82044a0b2ea64287f7f5337" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
I20260810 15:15:55.998535 1707 raft_consensus.cc:3037] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:55.998553 1413 raft_consensus.cc:3037] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Advancing to term 1
W20260810 15:15:55.999141 1534 tablet.cc:2369] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:55.999199 1707 raft_consensus.cc:2454] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:55.999374 1413 raft_consensus.cc:2454] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:55.999382 1849 leader_election.cc:302] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6b00349cc842efb1c563ab05973f64, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:55.999464 2102 raft_consensus.cc:2781] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:55.999646 2102 raft_consensus.cc:686] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [term 1 LEADER]: Becoming Leader. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Running, Role: LEADER
I20260810 15:15:55.999754 2102 consensus_queue.cc:237] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.000270 2139 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: TxnStatusTablet state changed. Reason: New leader b60d32461f56494fad52ea8e5a87087b. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.000375 2139 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:56.000566 2144 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:15:56.000562 1139 catalog_manager.cc:5495] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b reported cstate change: term changed from 0 to 1, leader changed from <none> to b60d32461f56494fad52ea8e5a87087b (127.29.230.3). New cstate: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } health_report { overall_health: HEALTHY } } }
I20260810 15:15:56.000748 2144 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:15:56.014389 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.011661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40360 (local address 127.29.230.3:34123)
0810 15:15:56.011708 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.011710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.011720 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:15:56.011798 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.011799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.011815 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.011869 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.011987 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.012480 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.013588 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.013802 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.013864 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.014048 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.014174 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.014229 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.014289 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.014309 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:15:56.014424 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.011565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.0.0.1:40360)
0810 15:15:56.011642 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:56.011657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:56.011711 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:56.011876 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:56.011877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:56.011878 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:56.011933 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.011936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.012553 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.012553 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.013480 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.013483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.013874 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.013875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.013944 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.013964 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.014227 (+ 263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.014234 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:15:56.014285 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:15:56.014338 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:15:56.016462 1707 raft_consensus.cc:1261] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.016502 1413 raft_consensus.cc:1261] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.016808 2139 consensus_queue.cc:1035] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:56.016966 2102 consensus_queue.cc:1035] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:56.018175 2101 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: TxnStatusTablet state changed. Reason: New leader b60d32461f56494fad52ea8e5a87087b. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.018275 2101 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:56.018317 2107 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: TxnStatusTablet state changed. Reason: New leader b60d32461f56494fad52ea8e5a87087b. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.018404 2107 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:56.019498 2102 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.019585 2102 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:56.019745 2102 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.019809 2102 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P b60d32461f56494fad52ea8e5a87087b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:15:56.020491 2107 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.020502 2101 tablet_replica.cc:439] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } } }
I20260810 15:15:56.020579 2107 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 3e6b00349cc842efb1c563ab05973f64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:56.020597 2101 tablet_replica.cc:442] T 28444dadb82044a0b2ea64287f7f5337 P 1b1e15b9c81440fd9903ea7a11f7c1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:15:56.021296 2153 mvcc.cc:205] Tried to move back new op lower bound from 7316991819840901120 to 7316991819775750144. Current Snapshot: MvccSnapshot[applied={T|T < 7316991819840901120}]
I20260810 15:15:56.024989 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.022189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.29.230.1:46073)
0810 15:15:56.022243 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.022245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.022252 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:15:56.022279 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.022279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.022295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.022352 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.022510 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.022978 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.024177 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.024387 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.024439 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.024652 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.024760 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.024832 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.024882 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.024934 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:15:56.033898 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.027478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53403 (local address 127.29.230.3:34123)
0810 15:15:56.027516 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.027518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.031501 (+ 3983us) server_negotiation.cc:407] Connection header received
0810 15:15:56.031560 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.031561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.031572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.031624 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.031750 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.032280 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.033080 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.033316 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.033383 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.033655 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.033696 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.033751 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.033819 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.033840 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:15:56.033951 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.027419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.29.230.1:53403)
0810 15:15:56.031419 (+ 4000us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:56.031432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:56.031504 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:56.031628 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:56.031629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:56.031630 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:56.031695 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.031697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.032382 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.032384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.033001 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.033003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.033398 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.033399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.033494 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.033513 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.033764 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.033766 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:15:56.033814 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:15:56.033865 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3957,"thread_start_us":82,"threads_started":1}
I20260810 15:15:56.036592 1707 raft_consensus.cc:1261] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Refusing update from remote peer 1b1e15b9c81440fd9903ea7a11f7c1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.036695 1970 raft_consensus.cc:1261] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Refusing update from remote peer 1b1e15b9c81440fd9903ea7a11f7c1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.036877 2111 consensus_queue.cc:1035] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:56.037009 2101 consensus_queue.cc:1035] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:56.038753 2155 mvcc.cc:205] Tried to move back new op lower bound from 7316991819923574784 to 7316991819452186624. Current Snapshot: MvccSnapshot[applied={T|T < 7316991819923574784}]
I20260810 15:15:56.040271 2150 mvcc.cc:205] Tried to move back new op lower bound from 7316991819923574784 to 7316991819452186624. Current Snapshot: MvccSnapshot[applied={T|T < 7316991819923574784}]
I20260810 15:15:56.045953 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.043189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46073 (local address 127.29.230.3:46079)
0810 15:15:56.043250 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:56.043275 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:56.043326 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:56.043450 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:56.043450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:56.043452 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:56.043530 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.043532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.044108 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.044109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.045080 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.045084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.045432 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.045432 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.045501 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.045517 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.045774 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.045776 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:15:56.045825 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 15:15:56.045873 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:15:56.045909 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.043223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:46079 (local address 127.29.230.1:46073)
0810 15:15:56.043258 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.043281 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.043312 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:15:56.043377 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.043378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.043387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.043422 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.043592 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.044040 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.045160 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.045362 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.045412 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.045637 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.045671 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.045758 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.045829 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.045854 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:15:56.059552 1138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:56.059672 1138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:56.060462 1138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Sending DeleteTablet for 3 replicas of tablet 19e0e27eda46404d9ddca27fc5562cd7
I20260810 15:15:56.060823 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:56 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:48758
I20260810 15:15:56.060910 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:56 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33606
I20260810 15:15:56.061192 2186 tablet_replica.cc:330] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: stopping tablet replica
I20260810 15:15:56.061339 2186 raft_consensus.cc:2229] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:56.061440 2186 raft_consensus.cc:2258] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:56.061409 1138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:15:56.061563 1138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:56.061736 1941 tablet_service.cc:1514] Processing DeleteTablet for tablet 19e0e27eda46404d9ddca27fc5562cd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:56 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:40348
I20260810 15:15:56.062027 2187 tablet_replica.cc:330] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: stopping tablet replica
I20260810 15:15:56.062125 2187 raft_consensus.cc:2229] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:56.062215 2187 raft_consensus.cc:2258] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:56.062698 2187 ts_tablet_manager.cc:1905] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:56.063366 1379 tablet_service.cc:1467] Processing CreateTablet for tablet e65df2b3908d4d648233a745364e7d99 (DEFAULT_TABLE table=test-table [id=4eeea624a760440e969ad367bf5b2a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:56.063477 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65df2b3908d4d648233a745364e7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:56.063468 1673 tablet_service.cc:1467] Processing CreateTablet for tablet e65df2b3908d4d648233a745364e7d99 (DEFAULT_TABLE table=test-table [id=4eeea624a760440e969ad367bf5b2a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:56.063601 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65df2b3908d4d648233a745364e7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:56.064769 1941 tablet_service.cc:1467] Processing CreateTablet for tablet e65df2b3908d4d648233a745364e7d99 (DEFAULT_TABLE table=test-table [id=4eeea624a760440e969ad367bf5b2a66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:56.064881 1941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e65df2b3908d4d648233a745364e7d99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:56.064915 2099 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Bootstrap starting.
I20260810 15:15:56.065351 2099 tablet_bootstrap.cc:654] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:56.065999 2099 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: No bootstrap required, opened a new log
I20260810 15:15:56.066051 2099 ts_tablet_manager.cc:1397] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:56.066183 2089 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Bootstrap starting.
I20260810 15:15:56.066186 2099 raft_consensus.cc:348] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.066248 2099 raft_consensus.cc:374] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:56.066274 2099 raft_consensus.cc:729] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Initialized, Role: FOLLOWER
I20260810 15:15:56.066323 2099 consensus_queue.cc:260] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.066412 2099 ts_tablet_manager.cc:1428] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:56.066586 2089 tablet_bootstrap.cc:654] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:56.066982 2087 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Bootstrap starting.
I20260810 15:15:56.067379 2189 tablet_replica.cc:330] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: stopping tablet replica
I20260810 15:15:56.067544 2189 raft_consensus.cc:2229] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:56.067654 2189 raft_consensus.cc:2258] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:56.067569 2087 tablet_bootstrap.cc:654] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:56.068254 2189 ts_tablet_manager.cc:1905] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:56.068477 2186 ts_tablet_manager.cc:1905] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:56.070084 2186 ts_tablet_manager.cc:1918] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:56.070163 2186 log.cc:1196] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/19e0e27eda46404d9ddca27fc5562cd7
I20260810 15:15:56.070310 2189 ts_tablet_manager.cc:1918] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:56.070408 2186 ts_tablet_manager.cc:1939] T 19e0e27eda46404d9ddca27fc5562cd7 P 3e6b00349cc842efb1c563ab05973f64: Deleting consensus metadata
I20260810 15:15:56.070429 2189 log.cc:1196] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/19e0e27eda46404d9ddca27fc5562cd7
I20260810 15:15:56.070751 2189 ts_tablet_manager.cc:1939] T 19e0e27eda46404d9ddca27fc5562cd7 P b60d32461f56494fad52ea8e5a87087b: Deleting consensus metadata
I20260810 15:15:56.070816 1117 catalog_manager.cc:4841] TS 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191): tablet 19e0e27eda46404d9ddca27fc5562cd7 (table test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]) successfully deleted
I20260810 15:15:56.071201 1118 catalog_manager.cc:4841] TS b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123): tablet 19e0e27eda46404d9ddca27fc5562cd7 (table test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]) successfully deleted
I20260810 15:15:56.072094 2187 ts_tablet_manager.cc:1918] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:56.072167 2187 log.cc:1196] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/19e0e27eda46404d9ddca27fc5562cd7
I20260810 15:15:56.072400 2187 ts_tablet_manager.cc:1939] T 19e0e27eda46404d9ddca27fc5562cd7 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting consensus metadata
I20260810 15:15:56.072814 2089 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: No bootstrap required, opened a new log
I20260810 15:15:56.072865 2089 ts_tablet_manager.cc:1397] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 15:15:56.073010 2089 raft_consensus.cc:348] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.073083 2089 raft_consensus.cc:374] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:56.073104 2089 raft_consensus.cc:729] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Initialized, Role: FOLLOWER
I20260810 15:15:56.073145 2089 consensus_queue.cc:260] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.073230 2089 ts_tablet_manager.cc:1428] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:56.073371 1118 catalog_manager.cc:4841] TS 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073): tablet 19e0e27eda46404d9ddca27fc5562cd7 (table test-table [id=da15dfbeb37046e383eb92a5e5fc1d8e]) successfully deleted
I20260810 15:15:56.073827 2087 tablet_bootstrap.cc:492] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: No bootstrap required, opened a new log
I20260810 15:15:56.073884 2087 ts_tablet_manager.cc:1397] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 15:15:56.073997 2087 raft_consensus.cc:348] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.074064 2087 raft_consensus.cc:374] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:56.074083 2087 raft_consensus.cc:729] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Initialized, Role: FOLLOWER
I20260810 15:15:56.074126 2087 consensus_queue.cc:260] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.074208 2087 ts_tablet_manager.cc:1428] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:56.297139 2102 raft_consensus.cc:482] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:56.297255 2102 raft_consensus.cc:504] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.297452 2102 leader_election.cc:288] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:56.297668 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65df2b3908d4d648233a745364e7d99" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" is_pre_election: true
I20260810 15:15:56.297683 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65df2b3908d4d648233a745364e7d99" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64" is_pre_election: true
I20260810 15:15:56.297780 1413 raft_consensus.cc:2454] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:56.297798 1707 raft_consensus.cc:2454] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:56.297960 1852 leader_election.cc:302] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:56.298123 2102 raft_consensus.cc:2781] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:56.298226 2102 raft_consensus.cc:482] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:56.298277 2102 raft_consensus.cc:3037] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:56.298951 2102 raft_consensus.cc:504] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.299074 2102 leader_election.cc:288] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Requested vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:56.299221 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65df2b3908d4d648233a745364e7d99" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
I20260810 15:15:56.299329 1413 raft_consensus.cc:3037] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:56.299317 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e65df2b3908d4d648233a745364e7d99" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64"
I20260810 15:15:56.299401 1707 raft_consensus.cc:3037] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:56.299983 1707 raft_consensus.cc:2454] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:56.300032 1413 raft_consensus.cc:2454] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:56.300168 1849 leader_election.cc:302] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6b00349cc842efb1c563ab05973f64, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:56.300256 2102 raft_consensus.cc:2781] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:56.300308 2102 raft_consensus.cc:686] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 1 LEADER]: Becoming Leader. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Running, Role: LEADER
I20260810 15:15:56.300357 2102 consensus_queue.cc:237] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:56.300922 1138 catalog_manager.cc:5495] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b reported cstate change: term changed from 0 to 1, leader changed from <none> to b60d32461f56494fad52ea8e5a87087b (127.29.230.3). New cstate: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } health_report { overall_health: HEALTHY } } }
W20260810 15:15:56.312204 2046 tablet.cc:2369] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:15:56.352550 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.350441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.29.230.3:34123)
0810 15:15:56.350488 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.350489 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.350495 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:15:56.350514 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.350514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.350526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.350573 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.350687 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.351179 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.352029 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.352232 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.352280 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.352376 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.352402 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.352432 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.352472 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.352489 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:15:56.356534 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.354608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:34609 (local address 127.29.230.3:34123)
0810 15:15:56.354648 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:56.354649 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:56.354657 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:15:56.354676 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:56.354676 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:56.354684 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:56.354723 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:56.354807 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.355228 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.355930 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.356125 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.356170 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.356345 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.356367 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.356398 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:56.356453 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:15:56.356464 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:15:56.356585 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:56.354501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.29.230.3:34609)
0810 15:15:56.354575 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:56.354586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:56.354629 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:56.354727 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:56.354728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:56.354729 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:56.354772 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.354773 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.355316 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.355317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.355863 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:56.355865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:56.356204 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:56.356206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:56.356252 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:56.356256 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:56.356409 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:56.356410 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:15:56.356449 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 15:15:56.356481 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 15:15:56.358403 1413 raft_consensus.cc:1261] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.358644 1707 raft_consensus.cc:1261] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:56.358614 2173 consensus_queue.cc:1035] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:56.358826 2173 consensus_queue.cc:1035] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:15:56.363142 2150 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 15:15:57.880846 1138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:57.880947 1138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:57.881793 1138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Sending DeleteTablet for 3 replicas of tablet e65df2b3908d4d648233a745364e7d99
I20260810 15:15:57.882066 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet e65df2b3908d4d648233a745364e7d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:48758
I20260810 15:15:57.882097 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet e65df2b3908d4d648233a745364e7d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33606
I20260810 15:15:57.882120 1941 tablet_service.cc:1514] Processing DeleteTablet for tablet e65df2b3908d4d648233a745364e7d99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:40348
I20260810 15:15:57.882409 2228 tablet_replica.cc:330] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: stopping tablet replica
I20260810 15:15:57.882493 2230 tablet_replica.cc:330] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: stopping tablet replica
I20260810 15:15:57.882546 2230 raft_consensus.cc:2229] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:57.882616 2228 raft_consensus.cc:2229] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:57.882658 2228 raft_consensus.cc:2258] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:57.882759 2230 raft_consensus.cc:2258] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:57.883028 2230 ts_tablet_manager.cc:1905] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.883056 2228 ts_tablet_manager.cc:1905] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.884769 2229 tablet_replica.cc:330] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: stopping tablet replica
I20260810 15:15:57.884881 2230 ts_tablet_manager.cc:1918] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:15:57.884881 2228 ts_tablet_manager.cc:1918] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:15:57.884895 2229 raft_consensus.cc:2229] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:57.884940 2228 log.cc:1196] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/e65df2b3908d4d648233a745364e7d99
I20260810 15:15:57.884989 2230 log.cc:1196] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/e65df2b3908d4d648233a745364e7d99
I20260810 15:15:57.884941 2229 raft_consensus.cc:2258] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:57.885217 2228 ts_tablet_manager.cc:1939] T e65df2b3908d4d648233a745364e7d99 P 3e6b00349cc842efb1c563ab05973f64: Deleting consensus metadata
I20260810 15:15:57.885251 2230 ts_tablet_manager.cc:1939] T e65df2b3908d4d648233a745364e7d99 P b60d32461f56494fad52ea8e5a87087b: Deleting consensus metadata
I20260810 15:15:57.885470 2229 ts_tablet_manager.cc:1905] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.885541 1117 catalog_manager.cc:4841] TS 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191): tablet e65df2b3908d4d648233a745364e7d99 (table test-table [id=4eeea624a760440e969ad367bf5b2a66]) successfully deleted
I20260810 15:15:57.885619 1118 catalog_manager.cc:4841] TS b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123): tablet e65df2b3908d4d648233a745364e7d99 (table test-table [id=4eeea624a760440e969ad367bf5b2a66]) successfully deleted
I20260810 15:15:57.886736 2229 ts_tablet_manager.cc:1918] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:15:57.886798 2229 log.cc:1196] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/e65df2b3908d4d648233a745364e7d99
I20260810 15:15:57.886989 2229 ts_tablet_manager.cc:1939] T e65df2b3908d4d648233a745364e7d99 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting consensus metadata
I20260810 15:15:57.887306 1118 catalog_manager.cc:4841] TS 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073): tablet e65df2b3908d4d648233a745364e7d99 (table test-table [id=4eeea624a760440e969ad367bf5b2a66]) successfully deleted
Aug 10 15:15:57 dist-test-slave-gjrz krb5kdc[1060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:15:57 dist-test-slave-gjrz krb5kdc[1060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:15:57.907877 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:57.901029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.29.230.62:36653)
0810 15:15:57.901171 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:57.901172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:57.901183 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:15:57.901209 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:57.901209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:57.901220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:57.901281 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:15:57.901809 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:57.902347 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:57.903112 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:57.903337 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:57.903385 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:57.905613 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:15:57.905615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:15:57.905615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:15:57.905622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:15:57.906909 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:57.907215 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:57.907216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:57.907216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:57.907298 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:15:57.907473 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:15:57.907473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:15:57.907474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:15:57.907582 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:15:57.907654 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:57.907731 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 15:15:57.907798 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260810 15:15:57.909308 1138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:15:57.909475 1138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:57.910780 1673 tablet_service.cc:1467] Processing CreateTablet for tablet cdde333ca1274de1858e733c4d8c6b05 (DEFAULT_TABLE table=test-table [id=4e0f390e84614aceb9906006a7fb8913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.910779 1379 tablet_service.cc:1467] Processing CreateTablet for tablet cdde333ca1274de1858e733c4d8c6b05 (DEFAULT_TABLE table=test-table [id=4e0f390e84614aceb9906006a7fb8913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.910782 1941 tablet_service.cc:1467] Processing CreateTablet for tablet cdde333ca1274de1858e733c4d8c6b05 (DEFAULT_TABLE table=test-table [id=4e0f390e84614aceb9906006a7fb8913]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.910918 1941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdde333ca1274de1858e733c4d8c6b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.910919 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdde333ca1274de1858e733c4d8c6b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.910928 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdde333ca1274de1858e733c4d8c6b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.912026 2245 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Bootstrap starting.
I20260810 15:15:57.912439 2245 tablet_bootstrap.cc:654] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.912447 2246 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Bootstrap starting.
I20260810 15:15:57.912477 2247 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Bootstrap starting.
I20260810 15:15:57.912799 2246 tablet_bootstrap.cc:654] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.912834 2247 tablet_bootstrap.cc:654] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.913352 2245 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: No bootstrap required, opened a new log
I20260810 15:15:57.913406 2245 ts_tablet_manager.cc:1397] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:57.913549 2247 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: No bootstrap required, opened a new log
I20260810 15:15:57.913532 2245 raft_consensus.cc:348] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.913599 2245 raft_consensus.cc:374] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.913610 2245 raft_consensus.cc:729] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.913614 2247 ts_tablet_manager.cc:1397] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:57.913659 2245 consensus_queue.cc:260] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.913719 2247 raft_consensus.cc:348] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.913791 2247 raft_consensus.cc:374] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.913810 2247 raft_consensus.cc:729] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.913821 2245 ts_tablet_manager.cc:1428] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:57.913889 2247 consensus_queue.cc:260] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.914088 2246 tablet_bootstrap.cc:492] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: No bootstrap required, opened a new log
I20260810 15:15:57.914095 2247 ts_tablet_manager.cc:1428] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:15:57.914165 2246 ts_tablet_manager.cc:1397] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:15:57.914355 2246 raft_consensus.cc:348] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.914476 2246 raft_consensus.cc:374] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.914517 2246 raft_consensus.cc:729] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.914562 2246 consensus_queue.cc:260] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.914709 2246 ts_tablet_manager.cc:1428] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:57.959829 2203 raft_consensus.cc:482] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:57.959932 2203 raft_consensus.cc:504] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.960135 2203 leader_election.cc:288] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:57.960382 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdde333ca1274de1858e733c4d8c6b05" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" is_pre_election: true
I20260810 15:15:57.960388 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdde333ca1274de1858e733c4d8c6b05" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64" is_pre_election: true
I20260810 15:15:57.960507 1707 raft_consensus.cc:2454] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:57.960507 1413 raft_consensus.cc:2454] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 0.
I20260810 15:15:57.960680 1852 leader_election.cc:302] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:57.960813 2203 raft_consensus.cc:2781] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:57.960856 2203 raft_consensus.cc:482] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:57.960876 2203 raft_consensus.cc:3037] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:57.961598 2203 raft_consensus.cc:504] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.961714 2203 leader_election.cc:288] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Requested vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191)
I20260810 15:15:57.961874 1707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdde333ca1274de1858e733c4d8c6b05" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6b00349cc842efb1c563ab05973f64"
I20260810 15:15:57.961874 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdde333ca1274de1858e733c4d8c6b05" candidate_uuid: "b60d32461f56494fad52ea8e5a87087b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
I20260810 15:15:57.961969 1707 raft_consensus.cc:3037] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:57.961984 1413 raft_consensus.cc:3037] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:57.962579 1413 raft_consensus.cc:2454] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:57.962585 1707 raft_consensus.cc:2454] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b60d32461f56494fad52ea8e5a87087b in term 1.
I20260810 15:15:57.962738 1849 leader_election.cc:302] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e6b00349cc842efb1c563ab05973f64, b60d32461f56494fad52ea8e5a87087b; no voters:
I20260810 15:15:57.962854 2203 raft_consensus.cc:2781] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:57.962914 2203 raft_consensus.cc:686] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 1 LEADER]: Becoming Leader. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Running, Role: LEADER
I20260810 15:15:57.962975 2203 consensus_queue.cc:237] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.963624 1139 catalog_manager.cc:5495] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b reported cstate change: term changed from 0 to 1, leader changed from <none> to b60d32461f56494fad52ea8e5a87087b (127.29.230.3). New cstate: current_term: 1 leader_uuid: "b60d32461f56494fad52ea8e5a87087b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } health_report { overall_health: HEALTHY } } }
I20260810 15:15:57.974956 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:57.972647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.29.230.3:34123)
0810 15:15:57.972814 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:57.972815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:57.972821 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:15:57.972838 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:57.972839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:57.972847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:57.972903 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:57.973049 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:57.973558 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:57.974314 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:57.974528 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:57.974582 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:57.974722 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:57.974759 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:57.974804 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:57.974855 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 15:15:57.974876 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260810 15:15:57.976933 1707 raft_consensus.cc:1261] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:57.976961 1413 raft_consensus.cc:1261] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Refusing update from remote peer b60d32461f56494fad52ea8e5a87087b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:57.977157 2203 consensus_queue.cc:1035] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:57.977264 2211 consensus_queue.cc:1035] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:57.987258 1139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:57.987377 1139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:57.988022 1139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Sending DeleteTablet for 3 replicas of tablet cdde333ca1274de1858e733c4d8c6b05
I20260810 15:15:57.988269 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet cdde333ca1274de1858e733c4d8c6b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:48758
I20260810 15:15:57.988335 1941 tablet_service.cc:1514] Processing DeleteTablet for tablet cdde333ca1274de1858e733c4d8c6b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:40348
I20260810 15:15:57.988382 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet cdde333ca1274de1858e733c4d8c6b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:57 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33606
I20260810 15:15:57.988473 2229 tablet_replica.cc:330] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: stopping tablet replica
I20260810 15:15:57.988476 2228 tablet_replica.cc:330] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: stopping tablet replica
I20260810 15:15:57.988528 2228 raft_consensus.cc:2229] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:57.988533 2229 raft_consensus.cc:2229] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:57.988473 1139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 15:15:57.988567 2229 raft_consensus.cc:2258] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:57.988567 2228 raft_consensus.cc:2258] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 15:15:57.988620 1139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:15:57.988641 2230 tablet_replica.cc:330] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: stopping tablet replica
I20260810 15:15:57.988700 2230 raft_consensus.cc:2229] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:57.988754 2230 raft_consensus.cc:2258] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:57.989042 2228 ts_tablet_manager.cc:1905] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.989274 2229 ts_tablet_manager.cc:1905] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.990723 2230 ts_tablet_manager.cc:1905] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:57.990748 2228 ts_tablet_manager.cc:1918] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:57.990748 2229 ts_tablet_manager.cc:1918] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:57.990803 2229 log.cc:1196] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/cdde333ca1274de1858e733c4d8c6b05
I20260810 15:15:57.990803 2228 log.cc:1196] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/cdde333ca1274de1858e733c4d8c6b05
I20260810 15:15:57.991015 2228 ts_tablet_manager.cc:1939] T cdde333ca1274de1858e733c4d8c6b05 P 3e6b00349cc842efb1c563ab05973f64: Deleting consensus metadata
I20260810 15:15:57.991022 2229 ts_tablet_manager.cc:1939] T cdde333ca1274de1858e733c4d8c6b05 P 1b1e15b9c81440fd9903ea7a11f7c1bc: Deleting consensus metadata
I20260810 15:15:57.991035 1379 tablet_service.cc:1467] Processing CreateTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f (DEFAULT_TABLE table=test-table [id=6beb6d56bdab4a41aaa247084bfb8a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.991137 1379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997dbb29b0a42b98161b62e0b1f5e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.992234 2230 ts_tablet_manager.cc:1918] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:15:57.992241 2247 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc: Bootstrap starting.
I20260810 15:15:57.992283 2230 log.cc:1196] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/cdde333ca1274de1858e733c4d8c6b05
I20260810 15:15:57.992319 1118 catalog_manager.cc:4841] TS 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073): tablet cdde333ca1274de1858e733c4d8c6b05 (table test-table [id=4e0f390e84614aceb9906006a7fb8913]) successfully deleted
I20260810 15:15:57.992478 2230 ts_tablet_manager.cc:1939] T cdde333ca1274de1858e733c4d8c6b05 P b60d32461f56494fad52ea8e5a87087b: Deleting consensus metadata
I20260810 15:15:57.992544 2247 tablet_bootstrap.cc:654] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.992542 1941 tablet_service.cc:1467] Processing CreateTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f (DEFAULT_TABLE table=test-table [id=6beb6d56bdab4a41aaa247084bfb8a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.992646 1941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997dbb29b0a42b98161b62e0b1f5e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.992865 1118 catalog_manager.cc:4841] TS b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123): tablet cdde333ca1274de1858e733c4d8c6b05 (table test-table [id=4e0f390e84614aceb9906006a7fb8913]) successfully deleted
I20260810 15:15:57.993139 2247 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc: No bootstrap required, opened a new log
I20260810 15:15:57.993184 2247 ts_tablet_manager.cc:1397] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:57.993276 2247 raft_consensus.cc:348] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.993340 2247 raft_consensus.cc:374] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.993352 2247 raft_consensus.cc:729] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1e15b9c81440fd9903ea7a11f7c1bc, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.993386 2247 consensus_queue.cc:260] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.993489 2247 ts_tablet_manager.cc:1428] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:57.993696 2246 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Bootstrap starting.
I20260810 15:15:57.993777 1117 catalog_manager.cc:4841] TS 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191): tablet cdde333ca1274de1858e733c4d8c6b05 (table test-table [id=4e0f390e84614aceb9906006a7fb8913]) successfully deleted
I20260810 15:15:57.993963 1673 tablet_service.cc:1467] Processing CreateTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f (DEFAULT_TABLE table=test-table [id=6beb6d56bdab4a41aaa247084bfb8a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:15:57.994014 2246 tablet_bootstrap.cc:654] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.994069 1673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c997dbb29b0a42b98161b62e0b1f5e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:15:57.994513 2246 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: No bootstrap required, opened a new log
I20260810 15:15:57.994551 2246 ts_tablet_manager.cc:1397] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:57.994637 2246 raft_consensus.cc:348] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.994699 2246 raft_consensus.cc:374] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.994711 2246 raft_consensus.cc:729] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60d32461f56494fad52ea8e5a87087b, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.994745 2246 consensus_queue.cc:260] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.994829 2246 ts_tablet_manager.cc:1428] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:57.995154 2245 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Bootstrap starting.
I20260810 15:15:57.995497 2245 tablet_bootstrap.cc:654] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Neither blocks nor log segments found. Creating new log.
I20260810 15:15:57.996001 2245 tablet_bootstrap.cc:492] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: No bootstrap required, opened a new log
I20260810 15:15:57.996042 2245 ts_tablet_manager.cc:1397] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:15:57.996161 2245 raft_consensus.cc:348] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.996214 2245 raft_consensus.cc:374] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:15:57.996230 2245 raft_consensus.cc:729] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Initialized, Role: FOLLOWER
I20260810 15:15:57.996271 2245 consensus_queue.cc:260] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:57.996356 2245 ts_tablet_manager.cc:1428] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:15:58.110886 2251 raft_consensus.cc:482] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:15:58.110988 2251 raft_consensus.cc:504] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:58.111304 2251 leader_election.cc:288] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123)
I20260810 15:15:58.114358 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.111547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:39045 (local address 127.29.230.1:46073)
0810 15:15:58.111659 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:58.111662 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:58.111671 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:58.111738 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:58.111739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:58.111749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:58.111829 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:58.112035 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.112550 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.113649 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.114049 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.114114 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.114124 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:58.114238 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 15:15:58.114280 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
I20260810 15:15:58.114367 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.111470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46073 (local address 127.29.230.2:39045)
0810 15:15:58.111616 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:58.111627 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:58.111680 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:58.111842 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:58.111843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:58.111845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:58.111936 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.111939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.112629 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.112630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.113569 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.113572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.114118 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.114118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.114179 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.114182 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:15:58.114227 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:15:58.114269 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":42,"thread_start_us":97,"threads_started":1}
I20260810 15:15:58.114460 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.111914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:51981 (local address 127.29.230.3:34123)
0810 15:15:58.111983 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:58.111985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:58.111994 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:15:58.112024 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:58.112025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:58.112036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:58.112083 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:15:58.112233 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.112693 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.113761 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.114138 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.114198 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.114206 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:58.114350 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:15:58.114392 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:15:58.114494 2276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.111727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.29.230.2:51981)
0810 15:15:58.111917 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:58.111927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:58.111974 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:58.112095 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:58.112096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:58.112098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:15:58.112181 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.112183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.112787 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.112788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.113674 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.113677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.114208 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.114209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.114287 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.114291 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:15:58.114337 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:15:58.114377 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260810 15:15:58.114674 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997dbb29b0a42b98161b62e0b1f5e3f" candidate_uuid: "3e6b00349cc842efb1c563ab05973f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" is_pre_election: true
I20260810 15:15:58.114770 1413 raft_consensus.cc:2454] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6b00349cc842efb1c563ab05973f64 in term 0.
I20260810 15:15:58.114753 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997dbb29b0a42b98161b62e0b1f5e3f" candidate_uuid: "3e6b00349cc842efb1c563ab05973f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60d32461f56494fad52ea8e5a87087b" is_pre_election: true
I20260810 15:15:58.114843 1970 raft_consensus.cc:2454] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e6b00349cc842efb1c563ab05973f64 in term 0.
I20260810 15:15:58.114953 1578 leader_election.cc:302] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, 3e6b00349cc842efb1c563ab05973f64; no voters:
I20260810 15:15:58.115111 2251 raft_consensus.cc:2781] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:15:58.115157 2251 raft_consensus.cc:482] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:15:58.115170 2251 raft_consensus.cc:3037] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:58.115901 2251 raft_consensus.cc:504] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:58.116055 2251 leader_election.cc:288] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [CANDIDATE]: Term 1 election: Requested vote from peers 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073), b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123)
I20260810 15:15:58.116225 1970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997dbb29b0a42b98161b62e0b1f5e3f" candidate_uuid: "3e6b00349cc842efb1c563ab05973f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b60d32461f56494fad52ea8e5a87087b"
I20260810 15:15:58.116207 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c997dbb29b0a42b98161b62e0b1f5e3f" candidate_uuid: "3e6b00349cc842efb1c563ab05973f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc"
I20260810 15:15:58.116291 1970 raft_consensus.cc:3037] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:58.116292 1413 raft_consensus.cc:3037] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:15:58.116879 1970 raft_consensus.cc:2454] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6b00349cc842efb1c563ab05973f64 in term 1.
I20260810 15:15:58.116894 1413 raft_consensus.cc:2454] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e6b00349cc842efb1c563ab05973f64 in term 1.
I20260810 15:15:58.117053 1578 leader_election.cc:302] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1e15b9c81440fd9903ea7a11f7c1bc, 3e6b00349cc842efb1c563ab05973f64; no voters:
I20260810 15:15:58.117156 2251 raft_consensus.cc:2781] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:15:58.117312 2251 raft_consensus.cc:686] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 1 LEADER]: Becoming Leader. State: Replica: 3e6b00349cc842efb1c563ab05973f64, State: Running, Role: LEADER
I20260810 15:15:58.117379 2251 consensus_queue.cc:237] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } }
I20260810 15:15:58.118106 1138 catalog_manager.cc:5495] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "3e6b00349cc842efb1c563ab05973f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e6b00349cc842efb1c563ab05973f64" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 41191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 } health_report { overall_health: UNKNOWN } } }
I20260810 15:15:58.145488 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.143019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.29.230.2:41191)
0810 15:15:58.143193 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:58.143195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:58.143203 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:15:58.143240 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:58.143241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:58.143304 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:58.143380 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:58.143478 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.143989 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.144760 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.145006 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.145062 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.145201 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:58.145312 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:58.145360 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:58.145392 (+ 32us) server_negotiation.cc:299] Negotiation successful
0810 15:15:58.145410 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260810 15:15:58.150348 2260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.148098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:47493 (local address 127.29.230.3:34123)
0810 15:15:58.148140 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:58.148143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:58.148194 (+ 51us) server_negotiation.cc:407] Connection header received
0810 15:15:58.148261 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:58.148261 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:58.148273 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:58.148320 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:58.148515 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.148973 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.149695 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.149883 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.149943 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.150149 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:58.150176 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:58.150207 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:58.150271 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 15:15:58.150291 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:15:58.150386 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.147968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:34123 (local address 127.29.230.2:47493)
0810 15:15:58.148136 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:58.148147 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:58.148207 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:58.148359 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:58.148367 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:58.148370 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:58.148433 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.148434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.149039 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.149040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.149628 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.149630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.150003 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.150004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.150060 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.150074 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:58.150226 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:58.150228 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:15:58.150268 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 15:15:58.150308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260810 15:15:58.152329 1970 raft_consensus.cc:1261] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Refusing update from remote peer 3e6b00349cc842efb1c563ab05973f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:58.152379 1413 raft_consensus.cc:1261] T c997dbb29b0a42b98161b62e0b1f5e3f P 1b1e15b9c81440fd9903ea7a11f7c1bc [term 1 FOLLOWER]: Refusing update from remote peer 3e6b00349cc842efb1c563ab05973f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:15:58.152541 2251 consensus_queue.cc:1035] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b60d32461f56494fad52ea8e5a87087b" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:58.152675 2279 consensus_queue.cc:1035] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1e15b9c81440fd9903ea7a11f7c1bc" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:15:58.159744 2282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.157416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:47501 (local address 127.29.230.2:41191)
0810 15:15:58.157452 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:15:58.157453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:15:58.157565 (+ 112us) server_negotiation.cc:407] Connection header received
0810 15:15:58.157621 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:15:58.157621 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:15:58.157631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:15:58.157671 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:15:58.157766 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.158254 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.159019 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.159330 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.159387 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.159528 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:15:58.159563 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:15:58.159603 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 15:15:58.159658 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:15:58.159679 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:15:58.159768 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:58.157366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:41191 (local address 127.29.230.3:47501)
0810 15:15:58.157500 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:58.157510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:15:58.157566 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:15:58.157676 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:15:58.157676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:15:58.157678 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:15:58.157729 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.157730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.158324 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.158325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.158949 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:15:58.158952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:15:58.159396 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:15:58.159397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:15:58.159458 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:15:58.159463 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:15:58.159613 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:15:58.159615 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:15:58.159655 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 15:15:58.159699 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":94,"threads_started":1}
W20260810 15:15:58.160177 2204 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 15:15:59.683728 1138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:59.683831 1138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:15:59.684749 1138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45086e201d56467a97e74130b8fa2f75: Sending DeleteTablet for 3 replicas of tablet c997dbb29b0a42b98161b62e0b1f5e3f
I20260810 15:15:59.685050 1379 tablet_service.cc:1514] Processing DeleteTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:59 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33606
I20260810 15:15:59.685040 1673 tablet_service.cc:1514] Processing DeleteTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:59 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:48758
I20260810 15:15:59.685139 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1274
I20260810 15:15:59.685232 1940 tablet_service.cc:1514] Processing DeleteTablet for tablet c997dbb29b0a42b98161b62e0b1f5e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:15:59 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:40348
I20260810 15:15:59.696524 2353 tablet_replica.cc:330] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: stopping tablet replica
I20260810 15:15:59.696687 2353 raft_consensus.cc:2229] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:15:59.696846 2353 raft_consensus.cc:2258] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:59.698248 2353 ts_tablet_manager.cc:1905] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:15:59.700376 2353 ts_tablet_manager.cc:1918] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:15:59.700438 2353 log.cc:1196] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/c997dbb29b0a42b98161b62e0b1f5e3f
I20260810 15:15:59.700634 2353 ts_tablet_manager.cc:1939] T c997dbb29b0a42b98161b62e0b1f5e3f P 3e6b00349cc842efb1c563ab05973f64: Deleting consensus metadata
I20260810 15:15:59.701263 1117 catalog_manager.cc:4841] TS 3e6b00349cc842efb1c563ab05973f64 (127.29.230.2:41191): tablet c997dbb29b0a42b98161b62e0b1f5e3f (table test-table [id=6beb6d56bdab4a41aaa247084bfb8a54]) successfully deleted
I20260810 15:15:59.703511 2354 tablet_replica.cc:330] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: stopping tablet replica
I20260810 15:15:59.703603 2354 raft_consensus.cc:2229] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:15:59.703648 2354 raft_consensus.cc:2258] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:15:59.703990 2354 ts_tablet_manager.cc:1905] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 15:15:59.705506 1118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:46073 (error 108)
I20260810 15:15:59.705958 2354 ts_tablet_manager.cc:1918] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 15:15:59.706024 2354 log.cc:1196] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/c997dbb29b0a42b98161b62e0b1f5e3f
I20260810 15:15:59.706229 2354 ts_tablet_manager.cc:1939] T c997dbb29b0a42b98161b62e0b1f5e3f P b60d32461f56494fad52ea8e5a87087b: Deleting consensus metadata
I20260810 15:15:59.706244 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1546
I20260810 15:15:59.721470 1118 catalog_manager.cc:4841] TS b60d32461f56494fad52ea8e5a87087b (127.29.230.3:34123): tablet c997dbb29b0a42b98161b62e0b1f5e3f (table test-table [id=6beb6d56bdab4a41aaa247084bfb8a54]) successfully deleted
I20260810 15:15:59.722544 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:15:59.705955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:46073 (local address 127.0.0.1:50818)
0810 15:15:59.706088 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:15:59.706138 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
W20260810 15:15:59.722735 1118 catalog_manager.cc:4584] TS 1b1e15b9c81440fd9903ea7a11f7c1bc (127.29.230.1:46073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c997dbb29b0a42b98161b62e0b1f5e3f: Network error: Client connection negotiation failed: client connection to 127.29.230.1:46073: connect: Connection refused (error 111)
I20260810 15:15:59.728888 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1818
I20260810 15:15:59.749313 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1088
2026-08-10T15:15:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5920 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:15:59 dist-test-slave-gjrz krb5kdc[2362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:15:59 dist-test-slave-gjrz krb5kdc[2362](info): set up 2 sockets
Aug 10 15:15:59 dist-test-slave-gjrz krb5kdc[2362](info): commencing operation
krb5kdc: starting...
W20260810 15:16:00.950695 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:01.072132 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:39087
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:39087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:01.154419 2390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:01.154649 2390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:01.154672 2390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:01.155810 2390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:01.155845 2390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:01.155858 2390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:01.155869 2390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:01.155880 2390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:01.157320 2390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:39087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:01.157527 2390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:01.157708 2390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:01.162767 2403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.162750 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.162760 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:01.163161 2390 server_base.cc:1034] running on GCE node
I20260810 15:16:01.163476 2390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:01.163775 2390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:01.164951 2390 hybrid_clock.cc:648] HybridClock initialized: now 1786374961164922 us; error 40 us; skew 500 ppm
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:01.170619 2390 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:01.171078 2390 webserver.cc:460] Webserver started at http://127.29.230.62:35049/ using document root <none> and password file <none>
I20260810 15:16:01.171344 2390 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:01.171393 2390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:01.171532 2390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:01.172609 2390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "88ccec7b70264c51a7b239abc229b2a7"
server_key_iv: "b0b61ceea01d7cb086abb5e266a6085d"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.172952 2390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "88ccec7b70264c51a7b239abc229b2a7"
server_key_iv: "b0b61ceea01d7cb086abb5e266a6085d"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.174460 2390 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:01.175215 2413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.175395 2390 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:01.175486 2390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "88ccec7b70264c51a7b239abc229b2a7"
server_key_iv: "b0b61ceea01d7cb086abb5e266a6085d"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.175551 2390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:01.201189 2390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:01.203547 2390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:01.203727 2390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:01.208314 2390 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:39087
I20260810 15:16:01.208355 2513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:39087 every 8 connection(s)
I20260810 15:16:01.208698 2390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:01.209645 2514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:01.212142 2514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf: Bootstrap starting.
I20260810 15:16:01.212749 2514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:01.213042 2514 log.cc:824] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:01.213829 2514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf: No bootstrap required, opened a new log
I20260810 15:16:01.215503 2514 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } }
I20260810 15:16:01.215654 2514 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:01.215677 2514 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8751eaa5e4d4f18b7c2f9f2fed39caf, State: Initialized, Role: FOLLOWER
I20260810 15:16:01.215818 2514 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } }
I20260810 15:16:01.215883 2514 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:01.215905 2514 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:01.215953 2514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:01.216480 2514 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } }
I20260810 15:16:01.216580 2514 leader_election.cc:302] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8751eaa5e4d4f18b7c2f9f2fed39caf; no voters:
I20260810 15:16:01.216729 2514 leader_election.cc:288] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:01.216780 2517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:01.216988 2514 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:01.216991 2517 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [term 1 LEADER]: Becoming Leader. State: Replica: a8751eaa5e4d4f18b7c2f9f2fed39caf, State: Running, Role: LEADER
I20260810 15:16:01.217134 2517 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } }
I20260810 15:16:01.217511 2518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } } }
I20260810 15:16:01.217602 2518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:01.217800 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2390
I20260810 15:16:01.217883 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:01.218062 30616 external_mini_cluster.cc:1428] Setting key a2e6c6515a0c667b8d981381e803988d
I20260810 15:16:01.218081 2524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:01.217528 2519 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8751eaa5e4d4f18b7c2f9f2fed39caf. Latest consensus state: current_term: 1 leader_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8751eaa5e4d4f18b7c2f9f2fed39caf" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39087 } } }
I20260810 15:16:01.218163 2519 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:01.218631 2524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:01.220379 2524 catalog_manager.cc:1347] Generated new cluster ID: 3b999f3e50b643b7bec995ded425c71c
I20260810 15:16:01.220425 2524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:01.229755 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.218525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.29.230.62:39087)
0810 15:16:01.218681 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:01.218691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:01.219023 (+ 332us) server_negotiation.cc:407] Connection header received
0810 15:16:01.219199 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:01.219206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:01.219283 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:01.219421 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:01.220199 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.221589 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.222675 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.223300 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.223389 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.226245 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:01.226249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:01.226249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:01.226263 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:01.228164 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.228560 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.228563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.228563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.228664 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.228854 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.228854 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.228854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.229138 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:01.229270 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:01.229389 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 15:16:01.229526 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260810 15:16:01.241307 2524 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:01.241832 2524 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:01.252645 2524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8751eaa5e4d4f18b7c2f9f2fed39caf: Generated new TSK 0
I20260810 15:16:01.252847 2524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:01.281984 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:01.364289 2563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:01.364488 2563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:01.364511 2563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:01.365653 2563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:01.365696 2563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:01.365721 2563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:01.367182 2563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:01.367436 2563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:01.367666 2563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:01.372573 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.372601 2572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.372570 2576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:01.372978 2563 server_base.cc:1034] running on GCE node
I20260810 15:16:01.373176 2563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:01.373373 2563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:01.374506 2563 hybrid_clock.cc:648] HybridClock initialized: now 1786374961374490 us; error 30 us; skew 500 ppm
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:01.376991 2563 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:01.377430 2563 webserver.cc:460] Webserver started at http://127.29.230.1:45613/ using document root <none> and password file <none>
I20260810 15:16:01.377647 2563 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:01.377696 2563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:01.377823 2563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:01.378906 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "e5ffd36d13474bf0aea943869c29dbcc"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "fe2be499ee104fce254ce2d7a5881d43"
server_key_iv: "b3da36450abe5c832709603331b2b7b1"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.379248 2563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "e5ffd36d13474bf0aea943869c29dbcc"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "fe2be499ee104fce254ce2d7a5881d43"
server_key_iv: "b3da36450abe5c832709603331b2b7b1"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.380584 2563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:01.381283 2586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.381449 2563 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:01.381516 2563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "e5ffd36d13474bf0aea943869c29dbcc"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "fe2be499ee104fce254ce2d7a5881d43"
server_key_iv: "b3da36450abe5c832709603331b2b7b1"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.381592 2563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:01.390240 2563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:01.392279 2563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:01.392410 2563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:01.392607 2563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:01.392899 2563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:01.392933 2563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.392961 2563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:01.392977 2563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.398380 2563 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:37503
I20260810 15:16:01.398445 2792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:37503 every 8 connection(s)
I20260810 15:16:01.398738 2563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:01.407970 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.399439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53795 (local address 127.29.230.62:39087)
0810 15:16:01.399498 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:01.399500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:01.399718 (+ 218us) server_negotiation.cc:407] Connection header received
0810 15:16:01.400235 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:01.400236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:01.400253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:01.400307 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:01.401243 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.401749 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.402750 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.402964 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.403022 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.405161 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:01.405163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:01.405163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:01.405170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:01.406521 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.406902 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.406903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.406903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.406951 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.407181 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.407183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.407183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.407346 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:01.407411 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:01.407746 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 15:16:01.407860 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:16:01.408099 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.399428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39087 (local address 127.29.230.1:53795)
0810 15:16:01.399591 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:01.399628 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:01.400122 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:01.400378 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:01.400383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:01.400945 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:01.401137 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.401142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.401833 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.401834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.402660 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.402662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.403035 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.403036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.403165 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.403611 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:01.403620 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:01.405073 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:01.406609 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.406616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.406617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.406842 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.406997 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.406998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.406998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.407094 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.407429 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:01.407438 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:01.407601 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 15:16:01.407710 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:16:01.407873 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260810 15:16:01.408241 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2563
I20260810 15:16:01.408320 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:01.408493 30616 external_mini_cluster.cc:1428] Setting key d401ceb3c43a65e40f66c8fd8fa23769
I20260810 15:16:01.408681 2794 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39087
I20260810 15:16:01.408792 2794 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:01.408977 2794 heartbeater.cc:511] Master 127.29.230.62:39087 requested a full tablet report, sending...
I20260810 15:16:01.409566 2452 ts_manager.cc:194] Registered new tserver with Master: e5ffd36d13474bf0aea943869c29dbcc (127.29.230.1:37503)
I20260810 15:16:01.410178 2452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:53795
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:01.432469 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:01.515321 2806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:01.515538 2806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:01.515561 2806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:01.516786 2806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:01.516832 2806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:01.516858 2806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:01.518532 2806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:01.518790 2806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:01.519011 2806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:01.524290 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.524364 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.524550 2815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:01.524570 2806 server_base.cc:1034] running on GCE node
I20260810 15:16:01.524885 2806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:01.525175 2806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:01.526336 2806 hybrid_clock.cc:648] HybridClock initialized: now 1786374961526319 us; error 33 us; skew 500 ppm
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:01.529312 2806 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:01.532965 2806 webserver.cc:460] Webserver started at http://127.29.230.2:42185/ using document root <none> and password file <none>
I20260810 15:16:01.533249 2806 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:01.533305 2806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:01.533447 2806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:01.534771 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "a252984ef40a46fab041f1d79f2865ff"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "e11cf5a969d54de608fa75bb613fd1fb"
server_key_iv: "798128077e7b3213104f7be5ee6509f7"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.535133 2806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "a252984ef40a46fab041f1d79f2865ff"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "e11cf5a969d54de608fa75bb613fd1fb"
server_key_iv: "798128077e7b3213104f7be5ee6509f7"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.536451 2806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:01.537133 2833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.537282 2806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:01.537369 2806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "a252984ef40a46fab041f1d79f2865ff"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "e11cf5a969d54de608fa75bb613fd1fb"
server_key_iv: "798128077e7b3213104f7be5ee6509f7"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.537438 2806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:01.565179 2806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:01.567488 2806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:01.567643 2806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:01.567888 2806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:01.568212 2806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:01.568245 2806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.568274 2806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:01.568291 2806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.573993 2806 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:39297
I20260810 15:16:01.574108 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:39297 every 8 connection(s)
I20260810 15:16:01.574383 2806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:01.577894 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2806
I20260810 15:16:01.577983 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:01.578186 30616 external_mini_cluster.cc:1428] Setting key cb36df8343ff67cc22d05f914b15fbd1
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:01.584331 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.575176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:42121 (local address 127.29.230.62:39087)
0810 15:16:01.575282 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:01.575285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:01.575449 (+ 164us) server_negotiation.cc:407] Connection header received
0810 15:16:01.575953 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:01.575954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:01.575965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:01.576016 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:01.577049 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.577569 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.578952 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.579175 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.579236 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.581498 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:01.581500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:01.581500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:01.581507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:01.582717 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.583305 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.583306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.583307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.583374 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.583659 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.583660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.583660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.583767 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:01.583817 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:01.584159 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 15:16:01.584253 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:16:01.584584 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.575145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39087 (local address 127.29.230.2:42121)
0810 15:16:01.575327 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:01.575371 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:01.575870 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:01.576077 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:01.576080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:01.576651 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:01.576852 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.576858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.577732 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.577734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.578883 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.578886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.579248 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.579249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.579427 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.579914 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:01.579923 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:01.581369 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:01.582811 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.582820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.582822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.583179 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.583444 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.583445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.583445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.583596 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.583825 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:01.583831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:01.584006 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 15:16:01.584139 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 15:16:01.584342 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260810 15:16:01.585173 3037 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39087
I20260810 15:16:01.585271 3037 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:01.585496 3037 heartbeater.cc:511] Master 127.29.230.62:39087 requested a full tablet report, sending...
I20260810 15:16:01.585899 2452 ts_manager.cc:194] Registered new tserver with Master: a252984ef40a46fab041f1d79f2865ff (127.29.230.2:39297)
I20260810 15:16:01.586416 2452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:42121
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:01.605683 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:01.687994 3047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:01.688207 3047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:01.688231 3047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:01.689373 3047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:01.689415 3047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:01.689440 3047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:01.690865 3047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:01.691099 3047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:01.691341 3047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:01.696040 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.696041 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:01.696036 3057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:01.696419 3047 server_base.cc:1034] running on GCE node
I20260810 15:16:01.696621 3047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:01.696847 3047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:01.697991 3047 hybrid_clock.cc:648] HybridClock initialized: now 1786374961697974 us; error 26 us; skew 500 ppm
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:01.700469 3047 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:01.700902 3047 webserver.cc:460] Webserver started at http://127.29.230.3:44727/ using document root <none> and password file <none>
I20260810 15:16:01.701109 3047 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:01.701157 3047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:01.701282 3047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:01.702353 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "25a8778f014b4a8b9cb7ad17caa06fd5"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "a9d4ceafc2547c4dd5a0df2869242507"
server_key_iv: "b14124b5a0973a2a75f457109df917cc"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.702690 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "25a8778f014b4a8b9cb7ad17caa06fd5"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "a9d4ceafc2547c4dd5a0df2869242507"
server_key_iv: "b14124b5a0973a2a75f457109df917cc"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.704023 3047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:01.704841 3071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.705017 3047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:01.705113 3047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "25a8778f014b4a8b9cb7ad17caa06fd5"
format_stamp: "Formatted at 2026-08-10 15:16:01 on dist-test-slave-gjrz"
server_key: "a9d4ceafc2547c4dd5a0df2869242507"
server_key_iv: "b14124b5a0973a2a75f457109df917cc"
server_key_version: "encryptionkey@0"
I20260810 15:16:01.705178 3047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:01.717517 3047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:01.719686 3047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:01.719817 3047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:01.720029 3047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:01.720357 3047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:01.720391 3047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.720420 3047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:01.720446 3047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:01.726228 3047 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:46275
I20260810 15:16:01.726293 3267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:46275 every 8 connection(s)
I20260810 15:16:01.726615 3047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:01.731186 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3047
I20260810 15:16:01.731290 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:01.731454 30616 external_mini_cluster.cc:1428] Setting key 83fee485e87e5667ff8af502430e0f2d
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[2362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374961, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:01.735706 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.727380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:57139 (local address 127.29.230.62:39087)
0810 15:16:01.727425 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:01.727427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:01.727665 (+ 238us) server_negotiation.cc:407] Connection header received
0810 15:16:01.728201 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:01.728202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:01.728215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:01.728288 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:01.729246 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.729755 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.730752 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.730938 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.730997 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.733265 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:01.733266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:01.733267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:01.733275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:01.734361 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.734728 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.734730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.734730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.734785 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:01.735056 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:01.735057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:01.735057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:01.735176 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:01.735233 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:01.735532 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 15:16:01.735616 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:01.735874 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:01.727361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39087 (local address 127.29.230.3:57139)
0810 15:16:01.727542 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:01.727580 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:01.728098 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:01.728354 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:01.728359 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:01.728929 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:01.729147 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.729152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.729844 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.729845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.730666 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:01.730669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:01.731012 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:01.731013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:01.731133 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:01.731654 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:01.731667 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:01.733162 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:01.734435 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.734442 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.734444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.734659 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.734840 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:01.734841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:01.734841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:01.734994 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:01.735233 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:01.735240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:01.735405 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 15:16:01.735500 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 15:16:01.735662 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260810 15:16:01.736389 3270 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39087
I20260810 15:16:01.736481 3270 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:01.736663 3270 heartbeater.cc:511] Master 127.29.230.62:39087 requested a full tablet report, sending...
I20260810 15:16:01.737192 2452 ts_manager.cc:194] Registered new tserver with Master: 25a8778f014b4a8b9cb7ad17caa06fd5 (127.29.230.3:46275)
I20260810 15:16:01.737586 2452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:57139
I20260810 15:16:01.743168 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:16:01.754891 2552 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:01.754032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.29.230.62:39087)
0810 15:16:01.754097 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:01.754099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:01.754127 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:16:01.754196 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:01.754197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:01.754207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:01.754257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:01.754816 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33526: BlockingRecv error: recv got EOF from 127.0.0.1:33526 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:01.755767 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2563
I20260810 15:16:01.784891 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2806
I20260810 15:16:01.796234 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3047
I20260810 15:16:01.809690 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2390
2026-08-10T15:16:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2053 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[3287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[3287](info): set up 2 sockets
Aug 10 15:16:01 dist-test-slave-gjrz krb5kdc[3287](info): commencing operation
krb5kdc: starting...
W20260810 15:16:02.998255 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:03.121718 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34009
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:03.203859 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:03.204092 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:03.204123 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:03.205296 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:03.205339 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:03.205358 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:03.205374 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:03.205389 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:03.206947 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:03.207203 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:03.207442 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:03.212466 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:03.212486 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:03.213632 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:03.213742 3305 server_base.cc:1034] running on GCE node
I20260810 15:16:03.213999 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:03.214310 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:03.215464 3305 hybrid_clock.cc:648] HybridClock initialized: now 1786374963215447 us; error 35 us; skew 500 ppm
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374963, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:03.217962 3305 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:03.218421 3305 webserver.cc:460] Webserver started at http://127.29.230.62:42203/ using document root <none> and password file <none>
I20260810 15:16:03.218634 3305 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:03.218683 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:03.218824 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:03.219952 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "d56aba015ac14d60ab71e78a3957b430"
format_stamp: "Formatted at 2026-08-10 15:16:03 on dist-test-slave-gjrz"
server_key: "4320982714ca07f69d2f3c73be7d67ec"
server_key_iv: "2bcd01b8bce3d45df3b8a11b4ff77d0f"
server_key_version: "encryptionkey@0"
I20260810 15:16:03.220293 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "d56aba015ac14d60ab71e78a3957b430"
format_stamp: "Formatted at 2026-08-10 15:16:03 on dist-test-slave-gjrz"
server_key: "4320982714ca07f69d2f3c73be7d67ec"
server_key_iv: "2bcd01b8bce3d45df3b8a11b4ff77d0f"
server_key_version: "encryptionkey@0"
I20260810 15:16:03.221568 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:03.222368 3329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:03.222582 3305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:03.222680 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "d56aba015ac14d60ab71e78a3957b430"
format_stamp: "Formatted at 2026-08-10 15:16:03 on dist-test-slave-gjrz"
server_key: "4320982714ca07f69d2f3c73be7d67ec"
server_key_iv: "2bcd01b8bce3d45df3b8a11b4ff77d0f"
server_key_version: "encryptionkey@0"
I20260810 15:16:03.222743 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:03.233620 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:03.235587 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:03.235723 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:03.239814 3305 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34009
I20260810 15:16:03.239861 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34009 every 8 connection(s)
I20260810 15:16:03.240386 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:03.241098 3411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:03.243453 3411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430: Bootstrap starting.
I20260810 15:16:03.244004 3411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:03.244257 3411 log.cc:824] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:03.244903 3411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430: No bootstrap required, opened a new log
I20260810 15:16:03.246462 3411 raft_consensus.cc:348] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } }
I20260810 15:16:03.246609 3411 raft_consensus.cc:374] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:03.246632 3411 raft_consensus.cc:729] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56aba015ac14d60ab71e78a3957b430, State: Initialized, Role: FOLLOWER
I20260810 15:16:03.246749 3411 consensus_queue.cc:260] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } }
I20260810 15:16:03.246825 3411 raft_consensus.cc:388] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:03.246853 3411 raft_consensus.cc:482] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:03.246886 3411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:03.247126 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3305
I20260810 15:16:03.247221 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:03.247432 30616 external_mini_cluster.cc:1428] Setting key 690ab20d3ee02ddcb705165994574dc6
I20260810 15:16:03.248095 3411 raft_consensus.cc:504] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } }
I20260810 15:16:03.248203 3411 leader_election.cc:302] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d56aba015ac14d60ab71e78a3957b430; no voters:
I20260810 15:16:03.248368 3411 leader_election.cc:288] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:03.248417 3419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:03.248631 3419 raft_consensus.cc:686] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [term 1 LEADER]: Becoming Leader. State: Replica: d56aba015ac14d60ab71e78a3957b430, State: Running, Role: LEADER
I20260810 15:16:03.248617 3411 sys_catalog.cc:564] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:03.248797 3419 consensus_queue.cc:237] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } }
I20260810 15:16:03.249419 3423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d56aba015ac14d60ab71e78a3957b430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } } }
I20260810 15:16:03.249526 3423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:03.249845 3422 sys_catalog.cc:455] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d56aba015ac14d60ab71e78a3957b430. Latest consensus state: current_term: 1 leader_uuid: "d56aba015ac14d60ab71e78a3957b430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d56aba015ac14d60ab71e78a3957b430" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34009 } } }
I20260810 15:16:03.249917 3438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:03.249993 3422 sys_catalog.cc:458] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:03.250547 3438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:03.252069 3438 catalog_manager.cc:1347] Generated new cluster ID: 9b313e0b6beb4e7fb64179cd36628fa2
I20260810 15:16:03.252122 3438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:03.259549 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:03.247937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.29.230.62:34009)
0810 15:16:03.248117 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:03.248126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:03.248152 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:03.248278 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:03.248285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:03.248338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:03.248486 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:03.249367 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:03.250776 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:03.251843 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:03.252440 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:03.252540 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:03.255238 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:03.255243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:03.255244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:03.255256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:03.257902 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:03.258276 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:03.258280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:03.258280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:03.258396 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:03.258576 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:03.258577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:03.258577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:03.258906 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:03.259069 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:03.259141 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 15:16:03.259316 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260810 15:16:03.260583 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:16:03.266465 3438 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:03.266979 3438 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:03.282821 3438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d56aba015ac14d60ab71e78a3957b430: Generated new TSK 0
I20260810 15:16:03.283057 3438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:03.286233 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:03.282274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46510 (local address 127.29.230.62:34009)
0810 15:16:03.282318 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:03.282320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:03.282484 (+ 164us) server_negotiation.cc:407] Connection header received
0810 15:16:03.282668 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:03.282669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:03.282681 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:03.282733 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:03.283597 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:03.284112 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:03.284830 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:03.285048 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:03.285105 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:03.285777 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:03.285778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:03.285779 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:03.285988 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:03.286006 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:03.286013 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:03.286062 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:03.286130 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 15:16:03.286159 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:03.288900 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3305
2026-08-10T15:16:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1474 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3476](info): set up 2 sockets
Aug 10 15:16:03 dist-test-slave-gjrz krb5kdc[3476](info): commencing operation
krb5kdc: starting...
W20260810 15:16:04.490041 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:04 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:04.611809 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36003
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:04.693852 3499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:04.694074 3499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:04.694095 3499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:04.695223 3499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:04.695288 3499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:04.695303 3499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:04.695317 3499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:04.695340 3499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:04.696823 3499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:04.697072 3499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:04.697280 3499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:04.701977 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:04.702036 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:04.702054 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:04.702253 3499 server_base.cc:1034] running on GCE node
I20260810 15:16:04.702610 3499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:04.702921 3499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:04.704092 3499 hybrid_clock.cc:648] HybridClock initialized: now 1786374964704077 us; error 38 us; skew 500 ppm
Aug 10 15:16:04 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:04.706630 3499 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:04.707126 3499 webserver.cc:460] Webserver started at http://127.29.230.62:38679/ using document root <none> and password file <none>
I20260810 15:16:04.707391 3499 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:04.707444 3499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:04.707573 3499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:04.708648 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "f09e3dbc214c45cfb51beb7602a34cab"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "ae7bb5a15ebdc22d571a8bfb2d5801bc"
server_key_iv: "8a734b5250797f2f0501467c9b77ab2c"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.709029 3499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "f09e3dbc214c45cfb51beb7602a34cab"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "ae7bb5a15ebdc22d571a8bfb2d5801bc"
server_key_iv: "8a734b5250797f2f0501467c9b77ab2c"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.710361 3499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:04.711170 3541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:04.711378 3499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:04.711475 3499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "f09e3dbc214c45cfb51beb7602a34cab"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "ae7bb5a15ebdc22d571a8bfb2d5801bc"
server_key_iv: "8a734b5250797f2f0501467c9b77ab2c"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.711544 3499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:04.727373 3499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:04.729584 3499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:04.729736 3499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:04.734053 3499 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36003
I20260810 15:16:04.734100 3626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36003 every 8 connection(s)
I20260810 15:16:04.734454 3499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:04.735682 3627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:04.737251 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3499
I20260810 15:16:04.737357 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:04.737519 30616 external_mini_cluster.cc:1428] Setting key 84519f8b7497e8077d30a1d107722b96
I20260810 15:16:04.738406 3627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab: Bootstrap starting.
I20260810 15:16:04.739159 3627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:04.739476 3627 log.cc:824] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:04.740218 3627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab: No bootstrap required, opened a new log
I20260810 15:16:04.742391 3627 raft_consensus.cc:348] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } }
I20260810 15:16:04.742573 3627 raft_consensus.cc:374] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:04.742614 3627 raft_consensus.cc:729] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09e3dbc214c45cfb51beb7602a34cab, State: Initialized, Role: FOLLOWER
I20260810 15:16:04.742756 3627 consensus_queue.cc:260] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } }
I20260810 15:16:04.742856 3627 raft_consensus.cc:388] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:04.742890 3627 raft_consensus.cc:482] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:04.742944 3627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:04.743795 3627 raft_consensus.cc:504] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } }
I20260810 15:16:04.743917 3627 leader_election.cc:302] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f09e3dbc214c45cfb51beb7602a34cab; no voters:
I20260810 15:16:04.744135 3627 leader_election.cc:288] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:04.744187 3637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:04.744356 3637 raft_consensus.cc:686] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [term 1 LEADER]: Becoming Leader. State: Replica: f09e3dbc214c45cfb51beb7602a34cab, State: Running, Role: LEADER
I20260810 15:16:04.744447 3627 sys_catalog.cc:564] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:04.744527 3637 consensus_queue.cc:237] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } }
Aug 10 15:16:04 dist-test-slave-gjrz krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:04.744912 3637 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f09e3dbc214c45cfb51beb7602a34cab. Latest consensus state: current_term: 1 leader_uuid: "f09e3dbc214c45cfb51beb7602a34cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } } }
I20260810 15:16:04.744987 3637 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:04.745195 3653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:04.745630 3642 sys_catalog.cc:455] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f09e3dbc214c45cfb51beb7602a34cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f09e3dbc214c45cfb51beb7602a34cab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36003 } } }
I20260810 15:16:04.745769 3642 sys_catalog.cc:458] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:04.746062 3653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:04.748731 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:04.738089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.29.230.62:36003)
0810 15:16:04.738302 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:04.738312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:04.738345 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:16:04.738905 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:04.738915 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:04.738979 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:04.739143 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:04.739832 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.740821 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.741904 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.742393 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.742496 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:04.745390 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:04.745394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:04.745395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:04.745408 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:04.747238 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:04.747628 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:04.747632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:04.747632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:04.747727 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:04.747917 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:04.747918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:04.747918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:04.748174 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:04.748313 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:04.748420 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 15:16:04.748557 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260810 15:16:04.752014 3681 mvcc.cc:205] Tried to move back new op lower bound from 7316991855603720192 to 7316991855594143744. Current Snapshot: MvccSnapshot[applied={T|T < 7316991855603720192}]
I20260810 15:16:04.752439 3653 catalog_manager.cc:1347] Generated new cluster ID: 48389e792bd74c7fb373abe4011b0eb9
I20260810 15:16:04.752497 3653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:04.757874 3653 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:04.758450 3653 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:04.760592 3653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f09e3dbc214c45cfb51beb7602a34cab: Generated new TSK 0
I20260810 15:16:04.760758 3653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:04.794123 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:04.914774 3690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:04.915045 3690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:04.915078 3690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:04.916783 3690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:04.916848 3690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:04.916886 3690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:04.918942 3690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:04.919210 3690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:04.919507 3690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:04.924943 3698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:04.925074 3690 server_base.cc:1034] running on GCE node
W20260810 15:16:04.924944 3699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:04.924973 3702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:04.925390 3690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:04.925650 3690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:04.926795 3690 hybrid_clock.cc:648] HybridClock initialized: now 1786374964926783 us; error 29 us; skew 500 ppm
Aug 10 15:16:04 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:04.932298 3690 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:04.932775 3690 webserver.cc:460] Webserver started at http://127.29.230.1:41609/ using document root <none> and password file <none>
I20260810 15:16:04.933005 3690 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:04.933051 3690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:04.933162 3690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:04.934233 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "0b8d47ed54554b10ac19070734cb86fb"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "21bc0027712097ccd5bce61444a8a638"
server_key_iv: "3a61664ce0f770759bd46eab8044a460"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.934568 3690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "0b8d47ed54554b10ac19070734cb86fb"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "21bc0027712097ccd5bce61444a8a638"
server_key_iv: "3a61664ce0f770759bd46eab8044a460"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.935865 3690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:16:04.936635 3715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:04.936787 3690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:04.936869 3690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "0b8d47ed54554b10ac19070734cb86fb"
format_stamp: "Formatted at 2026-08-10 15:16:04 on dist-test-slave-gjrz"
server_key: "21bc0027712097ccd5bce61444a8a638"
server_key_iv: "3a61664ce0f770759bd46eab8044a460"
server_key_version: "encryptionkey@0"
I20260810 15:16:04.936945 3690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:04.949133 3690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:04.951135 3690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:04.951287 3690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:04.951517 3690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:04.951836 3690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:04.951870 3690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:04.951893 3690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:04.951917 3690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:04.957479 3690 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:40585
I20260810 15:16:04.957537 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:40585 every 8 connection(s)
I20260810 15:16:04.957857 3690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:04.960775 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3690
I20260810 15:16:04.960886 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:04.961071 30616 external_mini_cluster.cc:1428] Setting key 0b962a0d5b0abde6ff96cc3e6e828c12
Aug 10 15:16:04 dist-test-slave-gjrz krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374964, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:04.968483 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:04.958600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53077 (local address 127.29.230.62:36003)
0810 15:16:04.958665 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:04.958668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:04.958860 (+ 192us) server_negotiation.cc:407] Connection header received
0810 15:16:04.959362 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:04.959363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:04.959374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:04.959425 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:04.960353 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.961070 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.962276 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.962567 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.962637 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:04.965150 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:04.965152 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:04.965152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:04.965160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:04.966735 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:04.967297 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:04.967299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:04.967299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:04.967369 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:04.967689 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:04.967690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:04.967690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:04.967844 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:04.967908 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:04.968307 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 15:16:04.968396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:16:04.968710 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:04.958586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36003 (local address 127.29.230.1:53077)
0810 15:16:04.958743 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:04.958781 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:04.959285 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:04.959482 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:04.959486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:04.960060 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:04.960265 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:04.960270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.961177 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.961178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:04.962179 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:04.962182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:04.962685 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:04.962686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:04.962815 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:04.963412 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:04.963424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:04.965049 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:04.966825 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:04.966833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:04.966835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:04.967172 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:04.967448 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:04.967449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:04.967449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:04.967599 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:04.967916 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:04.967924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:04.968130 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 15:16:04.968258 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 15:16:04.968461 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260810 15:16:04.969272 3916 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36003
I20260810 15:16:04.969384 3916 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:04.969590 3916 heartbeater.cc:511] Master 127.29.230.62:36003 requested a full tablet report, sending...
I20260810 15:16:04.970149 3568 ts_manager.cc:194] Registered new tserver with Master: 0b8d47ed54554b10ac19070734cb86fb (127.29.230.1:40585)
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:16:04.970782 3568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:53077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:04.988021 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:05.087951 3923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:05.088176 3923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:05.088199 3923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:05.089399 3923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:05.089444 3923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:05.089471 3923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:05.090956 3923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:05.091195 3923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:05.091456 3923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:05.096475 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:05.096486 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:05.096483 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:05.096709 3923 server_base.cc:1034] running on GCE node
I20260810 15:16:05.096928 3923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:05.097177 3923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:05.098343 3923 hybrid_clock.cc:648] HybridClock initialized: now 1786374965098324 us; error 36 us; skew 500 ppm
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:05.100915 3923 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:05.101377 3923 webserver.cc:460] Webserver started at http://127.29.230.2:37505/ using document root <none> and password file <none>
I20260810 15:16:05.101586 3923 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:05.101635 3923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:05.101768 3923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:05.102842 3923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "6235e8c07756496db9c60120be842268"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "016c7a1d5087c8b90fee3151a41f6446"
server_key_iv: "87fe898bd29df46e87b560a91a33f318"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.103183 3923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "6235e8c07756496db9c60120be842268"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "016c7a1d5087c8b90fee3151a41f6446"
server_key_iv: "87fe898bd29df46e87b560a91a33f318"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.104562 3923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:05.105417 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.105587 3923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:05.105675 3923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "6235e8c07756496db9c60120be842268"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "016c7a1d5087c8b90fee3151a41f6446"
server_key_iv: "87fe898bd29df46e87b560a91a33f318"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.105737 3923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:05.119410 3923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:05.121506 3923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:05.121647 3923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:05.121863 3923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:05.122177 3923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:05.122210 3923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.122239 3923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:05.122256 3923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.127923 3923 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:36439
I20260810 15:16:05.127985 4145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:36439 every 8 connection(s)
I20260810 15:16:05.128273 3923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:05.135150 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3923
I20260810 15:16:05.135249 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:05.135449 30616 external_mini_cluster.cc:1428] Setting key 2b4650377aade29325c41b7b8e354e6c
I20260810 15:16:05.137437 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:05.128962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:44341 (local address 127.29.230.62:36003)
0810 15:16:05.129020 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:05.129022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:05.129218 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:16:05.129763 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:05.129764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:05.129775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:05.129827 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:05.130724 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.131251 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.132321 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.132524 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.132588 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:05.134740 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:05.134742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:05.134742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:05.134750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:05.136155 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.136575 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.136576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.136577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.136627 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.136865 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.136866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.136866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.136964 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:05.137010 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:05.137299 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 15:16:05.137371 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:16:05.137662 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:05.128929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36003 (local address 127.29.230.2:44341)
0810 15:16:05.129102 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:05.129137 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:05.129664 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:05.129881 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:05.129884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:05.130440 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:05.130633 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:05.130638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.131398 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.131399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:05.132217 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:05.132219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.132607 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.132608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:05.132714 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:05.133187 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:05.133196 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:05.134647 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:05.136227 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:05.136235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:05.136238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:05.136509 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:05.136682 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:05.136683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:05.136683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:05.136813 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:05.137012 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:05.137017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:05.137175 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 15:16:05.137278 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 15:16:05.137438 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260810 15:16:05.138278 4147 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36003
I20260810 15:16:05.138370 4147 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:05.138554 4147 heartbeater.cc:511] Master 127.29.230.62:36003 requested a full tablet report, sending...
I20260810 15:16:05.138939 3568 ts_manager.cc:194] Registered new tserver with Master: 6235e8c07756496db9c60120be842268 (127.29.230.2:36439)
I20260810 15:16:05.139353 3568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:44341
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:05.162672 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:05.246891 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:05.247095 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:05.247118 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:05.248278 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:05.248319 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:05.248345 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:05.249795 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:05.249998 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:05.250197 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:05.255072 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:05.255107 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:05.255157 4156 server_base.cc:1034] running on GCE node
W20260810 15:16:05.255105 4162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:05.255453 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:05.255705 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:05.256856 4156 hybrid_clock.cc:648] HybridClock initialized: now 1786374965256840 us; error 34 us; skew 500 ppm
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:05.259363 4156 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:05.259775 4156 webserver.cc:460] Webserver started at http://127.29.230.3:46849/ using document root <none> and password file <none>
I20260810 15:16:05.260004 4156 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:05.260061 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:05.260190 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:05.261219 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "018f43eb6af5494995cff00d053f2813"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "a81fe82f537b1439c87fd0e960bdb573"
server_key_iv: "419f279b8a0f6ed4c297458cbc13532d"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.261549 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "018f43eb6af5494995cff00d053f2813"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "a81fe82f537b1439c87fd0e960bdb573"
server_key_iv: "419f279b8a0f6ed4c297458cbc13532d"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.262874 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:05.263729 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.263899 4156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:05.263993 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "018f43eb6af5494995cff00d053f2813"
format_stamp: "Formatted at 2026-08-10 15:16:05 on dist-test-slave-gjrz"
server_key: "a81fe82f537b1439c87fd0e960bdb573"
server_key_iv: "419f279b8a0f6ed4c297458cbc13532d"
server_key_version: "encryptionkey@0"
I20260810 15:16:05.264065 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:05.275480 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:05.277428 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:05.277557 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:05.277779 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:05.278102 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:05.278137 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.278167 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:05.278184 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:05.283807 4156 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:34745
I20260810 15:16:05.283874 4379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:34745 every 8 connection(s)
I20260810 15:16:05.284170 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:05.288398 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4156
I20260810 15:16:05.288488 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:05.288658 30616 external_mini_cluster.cc:1428] Setting key 8235c20579513e13e255fac34a979f59
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:05.293395 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:05.284847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:46205 (local address 127.29.230.62:36003)
0810 15:16:05.284914 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:05.284916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:05.285106 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:16:05.285614 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:05.285615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:05.285628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:05.285681 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:05.286644 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.287170 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.288245 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.288453 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.288512 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:05.290820 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:05.290821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:05.290822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:05.290829 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:05.292150 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.292565 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.292566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.292566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.292626 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.292816 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.292816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.292816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.292930 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:05.292976 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:05.293267 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 15:16:05.293331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:16:05.293587 4382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:05.284820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36003 (local address 127.29.230.3:46205)
0810 15:16:05.284985 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:05.285021 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:05.285510 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:05.285752 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:05.285756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:05.286340 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:05.286542 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:05.286547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.287289 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.287290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:05.288151 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:05.288154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.288528 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.288529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:05.288659 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:05.289232 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:05.289243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:05.290731 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:05.292238 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:05.292245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:05.292247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:05.292496 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:05.292671 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:05.292672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:05.292672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:05.292772 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:05.293002 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:05.293009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:05.293152 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 15:16:05.293244 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:16:05.293382 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260810 15:16:05.294034 4380 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36003
I20260810 15:16:05.294124 4380 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:05.294301 4380 heartbeater.cc:511] Master 127.29.230.62:36003 requested a full tablet report, sending...
I20260810 15:16:05.294691 3568 ts_manager.cc:194] Registered new tserver with Master: 018f43eb6af5494995cff00d053f2813 (127.29.230.3:34745)
I20260810 15:16:05.295063 3568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:46205
I20260810 15:16:05.300486 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[3476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374965, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:05.324898 3634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:05.317389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.29.230.62:36003)
0810 15:16:05.317507 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:05.317509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:05.317570 (+ 61us) server_negotiation.cc:407] Connection header received
0810 15:16:05.317649 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:05.317651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:05.317660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:05.317706 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:05.318316 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.318811 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.319728 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:05.319979 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:05.320056 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:05.322437 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:05.322439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:05.322439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:05.322455 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:05.323761 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.324128 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.324129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.324130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.324185 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:05.324351 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:05.324352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:05.324352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:05.324523 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:05.324621 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:05.324722 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 15:16:05.324804 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 15:16:05.325157 3568 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55480
I20260810 15:16:05.325591 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3690
I20260810 15:16:05.338241 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3923
I20260810 15:16:05.353381 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4156
I20260810 15:16:05.370543 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3499
2026-08-10T15:16:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2087 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[4397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[4397](info): set up 2 sockets
Aug 10 15:16:05 dist-test-slave-gjrz krb5kdc[4397](info): commencing operation
krb5kdc: starting...
W20260810 15:16:06.551214 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:06 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:06.673341 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35625
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:35625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:06.755165 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:06.755419 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:06.755442 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:06.756659 4426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:06.756707 4426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:06.756721 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:06.756731 4426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:06.756742 4426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:06.758306 4426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:35625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:06.758554 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:06.758782 4426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:06.763615 4432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:06.763729 4426 server_base.cc:1034] running on GCE node
W20260810 15:16:06.763818 4434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:06.763615 4436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:06.764111 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:06.764389 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:06.765539 4426 hybrid_clock.cc:648] HybridClock initialized: now 1786374966765521 us; error 33 us; skew 500 ppm
Aug 10 15:16:06 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:06.772891 4426 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:06.773391 4426 webserver.cc:460] Webserver started at http://127.29.230.62:34139/ using document root <none> and password file <none>
I20260810 15:16:06.773614 4426 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:06.773661 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:06.773783 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:06.774902 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "20e45d47acd44c4aaee634c31d33b3c0"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "a03a8a70c35429cc21d2f86735e36a96"
server_key_iv: "187c9b112905ad821271232ea8abdd48"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.775461 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "20e45d47acd44c4aaee634c31d33b3c0"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "a03a8a70c35429cc21d2f86735e36a96"
server_key_iv: "187c9b112905ad821271232ea8abdd48"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.777140 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:06.777926 4446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:06.778127 4426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:06.778228 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "20e45d47acd44c4aaee634c31d33b3c0"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "a03a8a70c35429cc21d2f86735e36a96"
server_key_iv: "187c9b112905ad821271232ea8abdd48"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.778291 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:06.793812 4426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:06.795918 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:06.796031 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:06.800002 4426 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:35625
I20260810 15:16:06.800052 4536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:35625 every 8 connection(s)
I20260810 15:16:06.800508 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:06.801240 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:06.803606 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0: Bootstrap starting.
I20260810 15:16:06.804184 4537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:06.804448 4537 log.cc:824] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:06.805130 4537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0: No bootstrap required, opened a new log
I20260810 15:16:06.806687 4537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } }
I20260810 15:16:06.806823 4537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:06.806856 4537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e45d47acd44c4aaee634c31d33b3c0, State: Initialized, Role: FOLLOWER
I20260810 15:16:06.806991 4537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } }
I20260810 15:16:06.807054 4537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:06.807085 4537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:06.807132 4537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:06.808024 4537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } }
I20260810 15:16:06.808125 4537 leader_election.cc:302] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20e45d47acd44c4aaee634c31d33b3c0; no voters:
I20260810 15:16:06.808280 4537 leader_election.cc:288] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:06.808346 4543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:06.808537 4537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:06.808547 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4426
I20260810 15:16:06.808561 4543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [term 1 LEADER]: Becoming Leader. State: Replica: 20e45d47acd44c4aaee634c31d33b3c0, State: Running, Role: LEADER
I20260810 15:16:06.808655 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:06.808743 4543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } }
I20260810 15:16:06.808830 30616 external_mini_cluster.cc:1428] Setting key 8a10a05ae97e03e60bf8d24d1fc940bc
I20260810 15:16:06.809151 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20e45d47acd44c4aaee634c31d33b3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } } }
I20260810 15:16:06.809154 4545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20e45d47acd44c4aaee634c31d33b3c0. Latest consensus state: current_term: 1 leader_uuid: "20e45d47acd44c4aaee634c31d33b3c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20e45d47acd44c4aaee634c31d33b3c0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35625 } } }
I20260810 15:16:06.809360 4545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:06.809346 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:06.809616 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:06.810189 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:06.813215 4558 catalog_manager.cc:1347] Generated new cluster ID: 7f1b79cfcbdd4370b714d32a26211a31
I20260810 15:16:06.813270 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:16:06 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:06.819095 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:06.809773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47596 (local address 127.29.230.62:35625)
0810 15:16:06.809948 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:06.809958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:06.809994 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:16:06.810120 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:06.810127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:06.810196 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:06.811235 (+ 1039us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:06.811357 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.812700 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.813567 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.814007 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.814100 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:06.816231 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:06.816235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:06.816235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:06.816247 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:06.817703 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:06.818019 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:06.818022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:06.818023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:06.818113 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:06.818284 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:06.818284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:06.818285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:06.818520 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:06.818669 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:06.818765 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:16:06.818900 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260810 15:16:06.826174 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:06.826664 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:06.828692 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20e45d47acd44c4aaee634c31d33b3c0: Generated new TSK 0
I20260810 15:16:06.828866 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:06.857752 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:06.938261 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:06.938510 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:06.938544 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:06.939711 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:06.939750 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:06.939785 4590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:06.941277 4590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:06.941517 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:06.941720 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:06.946656 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:06.946662 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:06.946669 4606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:06.947075 4590 server_base.cc:1034] running on GCE node
I20260810 15:16:06.947343 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:06.947589 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:06.948777 4590 hybrid_clock.cc:648] HybridClock initialized: now 1786374966948759 us; error 30 us; skew 500 ppm
Aug 10 15:16:06 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:06.951377 4590 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:06.951918 4590 webserver.cc:460] Webserver started at http://127.29.230.1:38331/ using document root <none> and password file <none>
I20260810 15:16:06.952163 4590 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:06.952214 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:06.952342 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:06.953400 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "1bbdcbef802e4000bf558f44e8723656"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "42641daf06f0c06637012ef8801c1556"
server_key_iv: "f6d89205aee554e4a1bcafde4108b577"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.953748 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "1bbdcbef802e4000bf558f44e8723656"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "42641daf06f0c06637012ef8801c1556"
server_key_iv: "f6d89205aee554e4a1bcafde4108b577"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.955022 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:06.955842 4622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:06.956051 4590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:06.956138 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "1bbdcbef802e4000bf558f44e8723656"
format_stamp: "Formatted at 2026-08-10 15:16:06 on dist-test-slave-gjrz"
server_key: "42641daf06f0c06637012ef8801c1556"
server_key_iv: "f6d89205aee554e4a1bcafde4108b577"
server_key_version: "encryptionkey@0"
I20260810 15:16:06.956208 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:06.964357 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:06.966228 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:06.966357 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:06.966560 4590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:06.966838 4590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:06.966871 4590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:06.966899 4590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:06.966923 4590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:06.972508 4590 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:41009
I20260810 15:16:06.972561 4823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:41009 every 8 connection(s)
I20260810 15:16:06.972851 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:06.972988 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4590
I20260810 15:16:06.973095 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:06.973305 30616 external_mini_cluster.cc:1428] Setting key 684e37852cdaea4c1d2b04d2aa363f7c
Aug 10 15:16:06 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374966, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:16:06.982980 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:06.973673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53489 (local address 127.29.230.62:35625)
0810 15:16:06.973715 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:06.973716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:06.973790 (+ 74us) server_negotiation.cc:407] Connection header received
0810 15:16:06.974716 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:06.974717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:06.974730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:06.974801 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:06.975427 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.976038 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.977395 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.977603 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.977658 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:06.980058 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:06.980060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:06.980060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:06.980069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:06.981590 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:06.982032 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:06.982033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:06.982034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:06.982094 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:06.982339 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:06.982340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:06.982340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:06.982480 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:06.982552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:06.982817 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 15:16:06.982907 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:06.983080 4830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:06.973527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35625 (local address 127.29.230.1:53489)
0810 15:16:06.973683 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:06.973725 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:06.974602 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:06.974857 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:06.974860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:06.975113 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:06.975341 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:06.975346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.976131 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.976133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:06.977301 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:06.977305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:06.977686 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:06.977687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:06.977833 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:06.978393 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:06.978404 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:06.979961 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:06.981666 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:06.981673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:06.981675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:06.981953 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:06.982172 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:06.982173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:06.982173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:06.982283 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:06.982547 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:06.982552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:06.982686 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 15:16:06.982766 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:16:06.982902 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260810 15:16:06.983596 4824 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35625
I20260810 15:16:06.983702 4824 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:06.983876 4824 heartbeater.cc:511] Master 127.29.230.62:35625 requested a full tablet report, sending...
I20260810 15:16:06.984387 4477 ts_manager.cc:194] Registered new tserver with Master: 1bbdcbef802e4000bf558f44e8723656 (127.29.230.1:41009)
I20260810 15:16:06.985092 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:53489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:07.000097 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:07.081284 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:07.081497 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:07.081523 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:07.082710 4836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:07.082752 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:07.082778 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:07.084292 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:07.084501 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:07.084725 4836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:07.089695 4844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:07.089700 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:07.089840 4836 server_base.cc:1034] running on GCE node
W20260810 15:16:07.089695 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:07.090140 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:07.090363 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:07.091524 4836 hybrid_clock.cc:648] HybridClock initialized: now 1786374967091502 us; error 32 us; skew 500 ppm
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:07.093974 4836 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:07.094374 4836 webserver.cc:460] Webserver started at http://127.29.230.2:35441/ using document root <none> and password file <none>
I20260810 15:16:07.094550 4836 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:07.094589 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:07.094682 4836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:07.095737 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "d7618abdda4c463599274ba743f6c6f0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "b9b638841330f4786580676c7c6e888d"
server_key_iv: "6709a91abd4acb00ed2a7599b016b52f"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.096050 4836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "d7618abdda4c463599274ba743f6c6f0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "b9b638841330f4786580676c7c6e888d"
server_key_iv: "6709a91abd4acb00ed2a7599b016b52f"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.097211 4836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:07.097877 4869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.098096 4836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:07.098181 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "d7618abdda4c463599274ba743f6c6f0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "b9b638841330f4786580676c7c6e888d"
server_key_iv: "6709a91abd4acb00ed2a7599b016b52f"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.098246 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:07.119025 4836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:07.120963 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:07.121094 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:07.122035 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:07.122388 4836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:07.122469 4836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.122531 4836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:07.122561 4836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.128067 4836 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:35571
I20260810 15:16:07.128116 5057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:35571 every 8 connection(s)
I20260810 15:16:07.128429 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:07.135394 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4836
I20260810 15:16:07.135505 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:07.135715 30616 external_mini_cluster.cc:1428] Setting key 939c12ae391ade524faa4d465644a2a7
I20260810 15:16:07.138233 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.129174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:48243 (local address 127.29.230.62:35625)
0810 15:16:07.129250 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:07.129252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:07.129392 (+ 140us) server_negotiation.cc:407] Connection header received
0810 15:16:07.130380 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:07.130381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:07.130396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:07.130455 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:07.131088 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.131676 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.132678 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.132870 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.132927 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.135066 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:07.135068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:07.135078 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:07.135087 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:07.136721 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.137146 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.137148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.137148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.137206 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.137423 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.137423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.137424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.137556 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:07.137629 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:07.138035 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 15:16:07.138143 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:16:07.138355 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.129101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35625 (local address 127.29.230.2:48243)
0810 15:16:07.129273 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:07.129311 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:07.130264 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:07.130528 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:07.130532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:07.130790 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:07.130992 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:07.130997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.131764 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.131765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:07.132594 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:07.132596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.132941 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.132942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:07.133059 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.133557 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:07.133571 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:07.134927 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:07.136795 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:07.136803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:07.136806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:07.137076 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:07.137261 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:07.137261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:07.137262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:07.137369 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:07.137664 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:07.137671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:07.137854 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 15:16:07.137977 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 15:16:07.138164 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260810 15:16:07.138916 5058 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35625
I20260810 15:16:07.139024 5058 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:07.139230 5058 heartbeater.cc:511] Master 127.29.230.62:35625 requested a full tablet report, sending...
I20260810 15:16:07.139643 4477 ts_manager.cc:194] Registered new tserver with Master: d7618abdda4c463599274ba743f6c6f0 (127.29.230.2:35571)
I20260810 15:16:07.140020 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:48243
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:07.162608 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:07.244946 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:07.245160 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:07.245190 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:07.246312 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:07.246352 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:07.246387 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:07.248026 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:07.248263 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:07.248479 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:07.253305 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:07.253306 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:07.253305 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:07.253690 5064 server_base.cc:1034] running on GCE node
I20260810 15:16:07.253875 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:07.254096 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:07.255237 5064 hybrid_clock.cc:648] HybridClock initialized: now 1786374967255225 us; error 27 us; skew 500 ppm
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:07.257756 5064 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:07.258229 5064 webserver.cc:460] Webserver started at http://127.29.230.3:35149/ using document root <none> and password file <none>
I20260810 15:16:07.258435 5064 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:07.258481 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:07.258599 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:07.259752 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "8a7baa0ac5734d039b6997aaabeface0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "9d3c36010cf4b196671fb1ea08f988eb"
server_key_iv: "2966c3335f12be3555d35d3aa54be101"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.260095 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "8a7baa0ac5734d039b6997aaabeface0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "9d3c36010cf4b196671fb1ea08f988eb"
server_key_iv: "2966c3335f12be3555d35d3aa54be101"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.261389 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:07.262174 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.262351 5064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:07.262449 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "8a7baa0ac5734d039b6997aaabeface0"
format_stamp: "Formatted at 2026-08-10 15:16:07 on dist-test-slave-gjrz"
server_key: "9d3c36010cf4b196671fb1ea08f988eb"
server_key_iv: "2966c3335f12be3555d35d3aa54be101"
server_key_version: "encryptionkey@0"
I20260810 15:16:07.262517 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:07.290624 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:07.292634 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:07.292775 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:07.293022 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:07.293339 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:07.293375 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.293403 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:07.293421 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:07.299099 5064 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:35099
I20260810 15:16:07.299160 5278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:35099 every 8 connection(s)
I20260810 15:16:07.299505 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:07.308286 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.300188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:43953 (local address 127.29.230.62:35625)
0810 15:16:07.300242 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:07.300244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:07.300408 (+ 164us) server_negotiation.cc:407] Connection header received
0810 15:16:07.301395 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:07.301397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:07.301411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:07.301463 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:07.302119 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.302613 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.303639 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.303831 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.303888 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.305901 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:07.305902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:07.305902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:07.305911 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:07.306977 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.307361 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.307362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.307362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.307421 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.307629 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.307630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.307630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.307744 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:07.307801 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:07.308055 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 15:16:07.308193 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:16:07.308401 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.300156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35625 (local address 127.29.230.3:43953)
0810 15:16:07.300295 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:07.300330 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:07.301287 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:07.301534 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:07.301537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:07.301800 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:07.302028 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:07.302034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.302694 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.302695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:07.303562 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:07.303565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.303919 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.303920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:07.304031 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.304481 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:07.304490 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:07.305811 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:07.307048 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:07.307055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:07.307057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:07.307302 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:07.307474 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:07.307475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:07.307475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:07.307573 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:07.307799 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:07.307806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:07.307936 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 15:16:07.308026 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:16:07.308186 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260810 15:16:07.308399 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5064
I20260810 15:16:07.308508 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:07.308696 30616 external_mini_cluster.cc:1428] Setting key b7161c2b26de9bbc4d359bc022d3a2c1
I20260810 15:16:07.309165 5279 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35625
I20260810 15:16:07.309268 5279 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:07.309460 5279 heartbeater.cc:511] Master 127.29.230.62:35625 requested a full tablet report, sending...
I20260810 15:16:07.309820 4477 ts_manager.cc:194] Registered new tserver with Master: 8a7baa0ac5734d039b6997aaabeface0 (127.29.230.3:35099)
I20260810 15:16:07.310189 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:43953
I20260810 15:16:07.320649 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:16:07.344188 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.336787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.29.230.1:41009)
0810 15:16:07.336980 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:07.336985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:07.336996 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:07.337024 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:07.337027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:07.337041 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:07.337113 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:07.337626 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.338405 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.339153 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.339513 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.339579 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.341689 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:07.341697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:07.341697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:07.341704 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:07.343152 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.343539 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.343541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.343542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.343607 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.343746 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.343747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.343747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.343870 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:07.343972 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:07.344026 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 15:16:07.344091 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[4397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:07.351301 4566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:07.345386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.29.230.62:35625)
0810 15:16:07.345452 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:07.345454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:07.345462 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:07.345486 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:07.345487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:07.345497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:07.345545 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:07.345956 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.346444 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.347078 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:07.347303 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:07.347361 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:07.349214 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:07.349215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:07.349216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:07.349223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:07.350283 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.350572 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.350573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.350573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.350624 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:07.350781 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:07.350782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:07.350783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:07.350928 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:07.351023 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:07.351116 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 15:16:07.351191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 15:16:07.351526 4477 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47606
I20260810 15:16:07.351835 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4590
I20260810 15:16:07.367486 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4836
I20260810 15:16:07.377810 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5064
I20260810 15:16:07.386677 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4426
2026-08-10T15:16:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2019 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[5302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[5302](info): set up 2 sockets
Aug 10 15:16:07 dist-test-slave-gjrz krb5kdc[5302](info): commencing operation
krb5kdc: starting...
W20260810 15:16:08.576576 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:08 dist-test-slave-gjrz krb5kdc[5302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:08.697496 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:39525
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:08.780357 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:08.780557 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:08.780579 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:08.781713 5324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:08.781754 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:08.781767 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:08.781780 5324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:08.781790 5324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:08.783375 5324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:39525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:08.783609 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:08.783836 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:08.788688 5334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:08.788729 5324 server_base.cc:1034] running on GCE node
W20260810 15:16:08.788699 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:08.788709 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:08.789206 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:08.789498 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:08.790656 5324 hybrid_clock.cc:648] HybridClock initialized: now 1786374968790641 us; error 29 us; skew 500 ppm
Aug 10 15:16:08 dist-test-slave-gjrz krb5kdc[5302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:08.793249 5324 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:08.793633 5324 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:08.793690 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:08.793815 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:08.794880 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "0b88d91d393e430e8cd4eebf14e6446f"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "d33dcec57a83779b6919bfea9120704d"
server_key_iv: "9c17c2bdfa110712387af72bc2eb93b6"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.795217 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "0b88d91d393e430e8cd4eebf14e6446f"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "d33dcec57a83779b6919bfea9120704d"
server_key_iv: "9c17c2bdfa110712387af72bc2eb93b6"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.796499 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:08.797257 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:08.797468 5324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:08.797561 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "0b88d91d393e430e8cd4eebf14e6446f"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "d33dcec57a83779b6919bfea9120704d"
server_key_iv: "9c17c2bdfa110712387af72bc2eb93b6"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.797626 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:08.812477 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:08.814954 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:08.815109 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:08.819310 5324 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:39525
I20260810 15:16:08.819337 5441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:39525 every 8 connection(s)
I20260810 15:16:08.819629 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:08.820756 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:08.822623 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5324
I20260810 15:16:08.822724 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:08.822903 30616 external_mini_cluster.cc:1428] Setting key f917e4ef50a95db1433395c0bb0a5a67
I20260810 15:16:08.823377 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: Bootstrap starting.
I20260810 15:16:08.824044 5442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:08.824363 5442 log.cc:824] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:08.825135 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: No bootstrap required, opened a new log
I20260810 15:16:08.827164 5442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } }
I20260810 15:16:08.827337 5442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:08.827389 5442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b88d91d393e430e8cd4eebf14e6446f, State: Initialized, Role: FOLLOWER
I20260810 15:16:08.827528 5442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } }
I20260810 15:16:08.827597 5442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:08.827620 5442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:08.827663 5442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:08.828289 5442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } }
I20260810 15:16:08.828392 5442 leader_election.cc:302] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b88d91d393e430e8cd4eebf14e6446f; no voters:
I20260810 15:16:08.828554 5442 leader_election.cc:288] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:08.828583 5452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:08.828683 5452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [term 1 LEADER]: Becoming Leader. State: Replica: 0b88d91d393e430e8cd4eebf14e6446f, State: Running, Role: LEADER
I20260810 15:16:08.828763 5452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } }
I20260810 15:16:08.828819 5442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:08.829084 5452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b88d91d393e430e8cd4eebf14e6446f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } } }
I20260810 15:16:08.829159 5452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:08.829299 5452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b88d91d393e430e8cd4eebf14e6446f. Latest consensus state: current_term: 1 leader_uuid: "0b88d91d393e430e8cd4eebf14e6446f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b88d91d393e430e8cd4eebf14e6446f" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 39525 } } }
I20260810 15:16:08.829353 5452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f [sys.catalog]: This master's current role is: LEADER
Aug 10 15:16:08 dist-test-slave-gjrz krb5kdc[5302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
W20260810 15:16:08.830821 5480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:08.830919 5480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:08.830974 5461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:08.831107 5461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:08.832659 5461 catalog_manager.cc:1347] Generated new cluster ID: 4f18d00a21424c4190883f6d1ccdb8ae
I20260810 15:16:08.832710 5461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:08.833954 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:08.823459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.29.230.62:39525)
0810 15:16:08.823680 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:08.823688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:08.823714 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:08.824170 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:08.824180 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:08.824245 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:08.824366 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:08.825067 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.826023 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.827090 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.827659 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.827771 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:08.830392 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:08.830396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:08.830396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:08.830409 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:08.832226 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:08.832639 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:08.832642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:08.832643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:08.832753 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:08.832989 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:08.832990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:08.832990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:08.833297 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:08.833433 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:08.833590 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 15:16:08.833740 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260810 15:16:08.843400 5461 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:08.844077 5461 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:08.846513 5461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: Generated new TSK 0
I20260810 15:16:08.846688 5461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:08.873926 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:08.955873 5489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:08.956123 5489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:08.956152 5489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:08.957297 5489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:08.957331 5489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:08.957356 5489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:08.958864 5489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:08.959072 5489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:08.959287 5489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:08.964166 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:08.964171 5505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:08.964205 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:08.964218 5489 server_base.cc:1034] running on GCE node
I20260810 15:16:08.964613 5489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:08.964854 5489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:08.966027 5489 hybrid_clock.cc:648] HybridClock initialized: now 1786374968966001 us; error 32 us; skew 500 ppm
Aug 10 15:16:08 dist-test-slave-gjrz krb5kdc[5302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:08.968573 5489 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:08.968909 5489 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:08.968961 5489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:08.969130 5489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:08.970198 5489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "622a42adbe59493c8cd5d11627f49434"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "b79721be15c3ce3e15706d2feb86dd4d"
server_key_iv: "f33399e3c896dd3315a6943a4e3aba94"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.970538 5489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "622a42adbe59493c8cd5d11627f49434"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "b79721be15c3ce3e15706d2feb86dd4d"
server_key_iv: "f33399e3c896dd3315a6943a4e3aba94"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.971838 5489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:08.972537 5513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:08.972700 5489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:08.972790 5489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "622a42adbe59493c8cd5d11627f49434"
format_stamp: "Formatted at 2026-08-10 15:16:08 on dist-test-slave-gjrz"
server_key: "b79721be15c3ce3e15706d2feb86dd4d"
server_key_iv: "f33399e3c896dd3315a6943a4e3aba94"
server_key_version: "encryptionkey@0"
I20260810 15:16:08.972851 5489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:08.980696 5489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:08.982713 5489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:08.982851 5489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:08.983090 5489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:08.983420 5489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:08.983456 5489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:08.983485 5489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:08.983503 5489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:08.989030 5489 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:33909
I20260810 15:16:08.989104 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:33909 every 8 connection(s)
I20260810 15:16:08.989336 5489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:16:08 dist-test-slave-gjrz krb5kdc[5302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374968, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:08.998077 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:08.990049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:52539 (local address 127.29.230.62:39525)
0810 15:16:08.990125 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:08.990127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:08.990274 (+ 147us) server_negotiation.cc:407] Connection header received
0810 15:16:08.990768 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:08.990769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:08.990782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:08.990835 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:08.991782 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.992298 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.993278 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.993496 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.993551 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:08.995629 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:08.995630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:08.995630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:08.995639 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:08.996798 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:08.997176 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:08.997177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:08.997178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:08.997235 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:08.997446 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:08.997447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:08.997447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:08.997569 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:08.997634 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:08.997893 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 15:16:08.997986 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:16:08.998250 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:08.989984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39525 (local address 127.29.230.1:52539)
0810 15:16:08.990158 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:08.990196 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:08.990674 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:08.990903 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:08.990907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:08.991162 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:08.991689 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:08.991695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.992384 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.992385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:08.993198 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:08.993200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:08.993572 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:08.993573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:08.993689 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:08.994190 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:08.994199 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:08.995538 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:08.996878 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:08.996885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:08.996887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:08.997110 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:08.997289 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:08.997290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:08.997291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:08.997391 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:08.997627 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:08.997634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:08.997777 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 15:16:08.997867 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:16:08.998037 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260810 15:16:08.998742 5709 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39525
I20260810 15:16:08.998827 5709 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:08.998986 5709 heartbeater.cc:511] Master 127.29.230.62:39525 requested a full tablet report, sending...
I20260810 15:16:08.999221 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5489
I20260810 15:16:08.999364 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:08.999562 30616 external_mini_cluster.cc:1428] Setting key 9dbd0b943fe9e4143f5a4705c1acf767
I20260810 15:16:08.999593 5378 ts_manager.cc:194] Registered new tserver with Master: 622a42adbe59493c8cd5d11627f49434 (127.29.230.1:33909)
I20260810 15:16:09.000308 5378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:52539
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:09.025692 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:09.108014 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:09.108219 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:09.108251 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:09.109360 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:09.109400 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:09.109436 5718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:09.110926 5718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:09.111172 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:09.111397 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:09.116086 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.116148 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.116086 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.116348 5718 server_base.cc:1034] running on GCE node
I20260810 15:16:09.116617 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:09.116833 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:09.118021 5718 hybrid_clock.cc:648] HybridClock initialized: now 1786374969118007 us; error 32 us; skew 500 ppm
Aug 10 15:16:09 dist-test-slave-gjrz krb5kdc[5302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374969, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:09.120594 5718 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:09.120913 5718 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:09.120958 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:09.121083 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:09.122162 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "ecdf4663f1914d348fd87b51b6ccef53"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c1fb3db7602f28391004fff49ed911fa"
server_key_iv: "1818254da514670f3f18f5266977458e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.122507 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "ecdf4663f1914d348fd87b51b6ccef53"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c1fb3db7602f28391004fff49ed911fa"
server_key_iv: "1818254da514670f3f18f5266977458e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.123842 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:09.124656 5737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.124842 5718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:09.124922 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "ecdf4663f1914d348fd87b51b6ccef53"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c1fb3db7602f28391004fff49ed911fa"
server_key_iv: "1818254da514670f3f18f5266977458e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.124992 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:09.151386 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:09.153674 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:09.153811 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:09.154073 5718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:09.154407 5718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:09.154441 5718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.154472 5718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:09.154489 5718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.160161 5718 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:38535
I20260810 15:16:09.160204 5948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:38535 every 8 connection(s)
I20260810 15:16:09.160476 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:09.160755 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5718
I20260810 15:16:09.160869 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:09.161128 30616 external_mini_cluster.cc:1428] Setting key ebd1179d4a0502133a2ed5deb4f33bd0
Aug 10 15:16:09 dist-test-slave-gjrz krb5kdc[5302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374969, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
I20260810 15:16:09.171031 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.161251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:60311 (local address 127.29.230.62:39525)
0810 15:16:09.161311 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.161313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.161578 (+ 265us) server_negotiation.cc:407] Connection header received
0810 15:16:09.162242 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.162243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.162253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.162290 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.163516 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.164030 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.165439 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.165629 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.165686 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.168042 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.168043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.168044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:09.168053 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.169637 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.170040 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.170041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.170041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.170100 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.170321 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.170322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.170322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.170462 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.170541 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.170843 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.170931 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:16:09.171135 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.161242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39525 (local address 127.29.230.2:60311)
0810 15:16:09.161434 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.161483 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.162129 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.162363 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.162367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.162714 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:09.163400 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.163407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.164130 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.164131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.165319 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.165323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.165712 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.165713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.165852 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.166369 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:09.166381 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:09.167948 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:09.169723 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:09.169730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:09.169732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:09.169960 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:09.170162 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:09.170163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:09.170163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:09.170265 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:09.170556 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:09.170562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:09.170690 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.170807 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.170944 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260810 15:16:09.171628 5949 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39525
I20260810 15:16:09.171725 5949 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:09.171861 5949 heartbeater.cc:511] Master 127.29.230.62:39525 requested a full tablet report, sending...
I20260810 15:16:09.172204 5378 ts_manager.cc:194] Registered new tserver with Master: ecdf4663f1914d348fd87b51b6ccef53 (127.29.230.2:38535)
I20260810 15:16:09.172555 5378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:60311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:09.190817 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:09.273950 5961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:09.274154 5961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:09.274176 5961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:09.275254 5961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:09.275324 5961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:09.275349 5961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:09.276882 5961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:39525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:09.277088 5961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:09.277278 5961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:09.281908 5974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.281917 5977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.281917 5975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.282298 5961 server_base.cc:1034] running on GCE node
I20260810 15:16:09.282522 5961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:09.282755 5961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:09.283946 5961 hybrid_clock.cc:648] HybridClock initialized: now 1786374969283898 us; error 33 us; skew 500 ppm
Aug 10 15:16:09 dist-test-slave-gjrz krb5kdc[5302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374969, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:09.286440 5961 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:09.286741 5961 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:09.286782 5961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:09.286871 5961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:09.287896 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "73e76306181f4ca7800af346ffaecfbb"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "081bd1afabaaab826f137ac496e4f02f"
server_key_iv: "8bd101e039b6eae3e6be11b7f6089b51"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.288198 5961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "73e76306181f4ca7800af346ffaecfbb"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "081bd1afabaaab826f137ac496e4f02f"
server_key_iv: "8bd101e039b6eae3e6be11b7f6089b51"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.289386 5961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:09.290071 5985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.290220 5961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:09.290302 5961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "73e76306181f4ca7800af346ffaecfbb"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "081bd1afabaaab826f137ac496e4f02f"
server_key_iv: "8bd101e039b6eae3e6be11b7f6089b51"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.290361 5961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:09.299903 5961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:09.302019 5961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:09.302156 5961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:09.302374 5961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:09.302680 5961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:09.302712 5961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.302740 5961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:09.302764 5961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.308461 5961 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:40679
I20260810 15:16:09.308507 6188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:40679 every 8 connection(s)
I20260810 15:16:09.308764 5961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:16:09 dist-test-slave-gjrz krb5kdc[5302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374969, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:09.316293 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5961
I20260810 15:16:09.316387 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:09.316615 30616 external_mini_cluster.cc:1428] Setting key 2231fb85818081a8453950eebcceda05
I20260810 15:16:09.317785 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.309489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:42131 (local address 127.29.230.62:39525)
0810 15:16:09.309565 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.309568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.309758 (+ 190us) server_negotiation.cc:407] Connection header received
0810 15:16:09.310247 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.310248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.310261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.310313 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.311295 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.311803 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.312803 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.313003 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.313067 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.315138 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.315140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.315140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:09.315148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.316345 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.316839 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.316840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.316840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.316908 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.317171 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.317171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.317172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.317314 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.317380 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.317656 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.317722 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:16:09.317951 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.309420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:39525 (local address 127.29.230.3:42131)
0810 15:16:09.309633 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.309673 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.310154 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.310383 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.310394 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.310654 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:09.311175 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.311181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.311888 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.311889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.312713 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.312715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.313075 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.313076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.313203 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.313642 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:09.313651 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:09.315042 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:09.316455 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:09.316462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:09.316465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:09.316749 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:09.316973 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:09.316974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:09.316974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:09.317103 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:09.317374 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:09.317379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:09.317528 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.317633 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.317760 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260810 15:16:09.318383 6190 heartbeater.cc:347] Connected to a master server at 127.29.230.62:39525
I20260810 15:16:09.318481 6190 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:09.318627 6190 heartbeater.cc:511] Master 127.29.230.62:39525 requested a full tablet report, sending...
I20260810 15:16:09.318967 5378 ts_manager.cc:194] Registered new tserver with Master: 73e76306181f4ca7800af346ffaecfbb (127.29.230.3:40679)
I20260810 15:16:09.319356 5378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:42131
I20260810 15:16:09.328606 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:16:09.335606 5448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.330008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.29.230.62:39525)
0810 15:16:09.330078 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.330080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.330156 (+ 76us) server_negotiation.cc:407] Connection header received
0810 15:16:09.330208 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.330209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.330220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.330268 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.330843 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.331380 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.332074 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.332259 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.332306 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.333405 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.333406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.333406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:09.333412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.334598 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.334886 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.334886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.334886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.334932 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:09.335048 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:09.335048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:09.335049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:09.335183 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.335257 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.335441 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.335509 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 15:16:09.337101 5378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:09.338651 5378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:09.345109 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.341096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40679 (local address 127.0.0.1:46988)
0810 15:16:09.341281 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.341289 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.341325 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.341798 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.341799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.341801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:09.341847 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.341849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.343059 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.343060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.343989 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.343992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.344836 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.344837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.344908 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.344920 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.344974 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.345025 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":46,"threads_started":1}
I20260810 15:16:09.345120 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.341281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.29.230.3:40679)
0810 15:16:09.341444 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.341448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.341479 (+ 31us) server_negotiation.cc:407] Connection header received
0810 15:16:09.341685 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.341688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.341706 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.341929 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:09.342035 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.342967 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.344065 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.344761 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.344838 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.344919 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.344988 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.345033 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260810 15:16:09.345551 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.340985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.29.230.1:33909)
0810 15:16:09.341526 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.341530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.341540 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:09.341599 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.341601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.341615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.341685 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:09.341912 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.342601 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.344213 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.344937 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.345002 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.345055 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.345449 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.345491 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":85,"threads_started":1}
I20260810 15:16:09.345757 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.340848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:33909 (local address 127.0.0.1:40228)
0810 15:16:09.341146 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.341156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.341207 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.341771 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.341772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.341774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:09.341843 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.341845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.342691 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.342692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.344126 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.344128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.345045 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.345046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.345128 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.345131 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.345185 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.345660 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":77,"thread_start_us":57,"threads_started":1}
I20260810 15:16:09.346117 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.340844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:38535 (local address 127.0.0.1:54442)
0810 15:16:09.341009 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.341048 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.341113 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.341367 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.341371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.341373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:09.341448 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.341449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.342774 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.342775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.344120 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.344124 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.345897 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.345898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.345962 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.345965 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.346009 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.346051 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":95,"thread_start_us":88,"threads_started":1}
I20260810 15:16:09.346068 6073 tablet_service.cc:1467] Processing CreateTablet for tablet 90e470045dea4822ad48b7688dc44452 (DEFAULT_TABLE table=test-table [id=ee3623d5b11d4844971dfdbfe9c95d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:09.346390 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.340965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.29.230.2:38535)
0810 15:16:09.341197 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.341202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.341211 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:09.341249 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.341251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.341266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.341354 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:09.341561 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.342682 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.345091 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.345736 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.345804 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.345863 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.346265 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.346310 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260810 15:16:09.346449 6073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e470045dea4822ad48b7688dc44452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:09.346585 5591 tablet_service.cc:1467] Processing CreateTablet for tablet 90e470045dea4822ad48b7688dc44452 (DEFAULT_TABLE table=test-table [id=ee3623d5b11d4844971dfdbfe9c95d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:09.346895 5591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e470045dea4822ad48b7688dc44452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:09.346964 5830 tablet_service.cc:1467] Processing CreateTablet for tablet 90e470045dea4822ad48b7688dc44452 (DEFAULT_TABLE table=test-table [id=ee3623d5b11d4844971dfdbfe9c95d17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:09.347244 5830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e470045dea4822ad48b7688dc44452. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:09.349749 6208 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Bootstrap starting.
I20260810 15:16:09.349985 6209 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Bootstrap starting.
I20260810 15:16:09.350152 6211 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Bootstrap starting.
I20260810 15:16:09.350435 6208 tablet_bootstrap.cc:654] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:09.350685 6209 tablet_bootstrap.cc:654] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:09.350723 6211 tablet_bootstrap.cc:654] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:09.351043 6208 log.cc:824] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:09.351296 6211 log.cc:824] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:09.351296 6209 log.cc:824] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:09.351809 6208 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: No bootstrap required, opened a new log
I20260810 15:16:09.351884 6208 ts_tablet_manager.cc:1397] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:09.351972 6211 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: No bootstrap required, opened a new log
I20260810 15:16:09.351986 6209 tablet_bootstrap.cc:492] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: No bootstrap required, opened a new log
I20260810 15:16:09.352038 6211 ts_tablet_manager.cc:1397] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:09.352063 6209 ts_tablet_manager.cc:1397] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:09.353329 6211 raft_consensus.cc:348] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.353448 6211 raft_consensus.cc:374] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:09.353467 6211 raft_consensus.cc:729] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 622a42adbe59493c8cd5d11627f49434, State: Initialized, Role: FOLLOWER
I20260810 15:16:09.353566 6211 consensus_queue.cc:260] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.353829 5709 heartbeater.cc:503] Master 127.29.230.62:39525 was elected leader, sending a full tablet report...
I20260810 15:16:09.353811 6208 raft_consensus.cc:348] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.353961 6208 raft_consensus.cc:374] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:09.353996 6208 raft_consensus.cc:729] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecdf4663f1914d348fd87b51b6ccef53, State: Initialized, Role: FOLLOWER
I20260810 15:16:09.353827 6211 ts_tablet_manager.cc:1428] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:09.354139 6208 consensus_queue.cc:260] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.354277 6209 raft_consensus.cc:348] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.354426 6208 ts_tablet_manager.cc:1428] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:16:09.354452 6209 raft_consensus.cc:374] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:09.354501 6209 raft_consensus.cc:729] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e76306181f4ca7800af346ffaecfbb, State: Initialized, Role: FOLLOWER
I20260810 15:16:09.354547 5949 heartbeater.cc:503] Master 127.29.230.62:39525 was elected leader, sending a full tablet report...
I20260810 15:16:09.354609 6209 consensus_queue.cc:260] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.354866 6209 ts_tablet_manager.cc:1428] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:16:09.354910 6190 heartbeater.cc:503] Master 127.29.230.62:39525 was elected leader, sending a full tablet report...
W20260810 15:16:09.411335 5950 tablet.cc:2369] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:16:09.490070 5711 tablet.cc:2369] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:09.505662 6217 raft_consensus.cc:482] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:09.505810 6217 raft_consensus.cc:504] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.506183 6217 leader_election.cc:288] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 622a42adbe59493c8cd5d11627f49434 (127.29.230.1:33909), 73e76306181f4ca7800af346ffaecfbb (127.29.230.3:40679)
I20260810 15:16:09.508852 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.506435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:32801 (local address 127.29.230.1:33909)
0810 15:16:09.506495 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.506498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.506507 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:09.506539 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.506540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.506549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.506597 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:09.506718 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.507191 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508169 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.508546 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508611 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.508621 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.508748 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.508786 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:09.508852 5952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.506365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:33909 (local address 127.29.230.2:32801)
0810 15:16:09.506423 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.506434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.506488 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.506606 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.506607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.506608 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:09.506663 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.506665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.507302 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.507303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.508075 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.508077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.508613 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.508682 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.508685 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.508735 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.508777 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:16:09.508953 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.506365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40679 (local address 127.29.230.2:46437)
0810 15:16:09.506552 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.506561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.506620 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.506737 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.506737 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.506739 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:09.506805 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.506806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.507437 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.507438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.508186 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:09.508188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.508716 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:09.508785 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.508789 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.508833 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.508872 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":15,"thread_start_us":103,"threads_started":1}
I20260810 15:16:09.508942 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.506450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:46437 (local address 127.29.230.3:40679)
0810 15:16:09.506498 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.506500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.506618 (+ 118us) server_negotiation.cc:407] Connection header received
0810 15:16:09.506674 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.506674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.506684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.506729 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:09.506858 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.507367 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508265 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.508644 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.508705 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.508714 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.508844 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.508878 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:16:09.509229 5625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e470045dea4822ad48b7688dc44452" candidate_uuid: "ecdf4663f1914d348fd87b51b6ccef53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a42adbe59493c8cd5d11627f49434" is_pre_election: true
I20260810 15:16:09.509254 6108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e470045dea4822ad48b7688dc44452" candidate_uuid: "ecdf4663f1914d348fd87b51b6ccef53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e76306181f4ca7800af346ffaecfbb" is_pre_election: true
I20260810 15:16:09.509390 6108 raft_consensus.cc:2454] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecdf4663f1914d348fd87b51b6ccef53 in term 0.
I20260810 15:16:09.509390 5625 raft_consensus.cc:2454] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecdf4663f1914d348fd87b51b6ccef53 in term 0.
I20260810 15:16:09.509618 5738 leader_election.cc:302] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e76306181f4ca7800af346ffaecfbb, ecdf4663f1914d348fd87b51b6ccef53; no voters:
I20260810 15:16:09.509734 6217 raft_consensus.cc:2781] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:09.509793 6217 raft_consensus.cc:482] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:09.509810 6217 raft_consensus.cc:3037] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:09.510473 6217 raft_consensus.cc:504] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.510605 6217 leader_election.cc:288] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [CANDIDATE]: Term 1 election: Requested vote from peers 622a42adbe59493c8cd5d11627f49434 (127.29.230.1:33909), 73e76306181f4ca7800af346ffaecfbb (127.29.230.3:40679)
I20260810 15:16:09.510761 6108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e470045dea4822ad48b7688dc44452" candidate_uuid: "ecdf4663f1914d348fd87b51b6ccef53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73e76306181f4ca7800af346ffaecfbb"
I20260810 15:16:09.510838 6108 raft_consensus.cc:3037] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:09.511344 5625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e470045dea4822ad48b7688dc44452" candidate_uuid: "ecdf4663f1914d348fd87b51b6ccef53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "622a42adbe59493c8cd5d11627f49434"
I20260810 15:16:09.511427 5625 raft_consensus.cc:3037] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:09.511582 6108 raft_consensus.cc:2454] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecdf4663f1914d348fd87b51b6ccef53 in term 1.
I20260810 15:16:09.511752 5738 leader_election.cc:302] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73e76306181f4ca7800af346ffaecfbb, ecdf4663f1914d348fd87b51b6ccef53; no voters:
I20260810 15:16:09.511840 6217 raft_consensus.cc:2781] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:09.512003 6217 raft_consensus.cc:686] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 1 LEADER]: Becoming Leader. State: Replica: ecdf4663f1914d348fd87b51b6ccef53, State: Running, Role: LEADER
I20260810 15:16:09.512167 6217 consensus_queue.cc:237] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } }
I20260810 15:16:09.512269 5625 raft_consensus.cc:2454] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecdf4663f1914d348fd87b51b6ccef53 in term 1.
I20260810 15:16:09.513046 5377 catalog_manager.cc:5495] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecdf4663f1914d348fd87b51b6ccef53 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "ecdf4663f1914d348fd87b51b6ccef53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecdf4663f1914d348fd87b51b6ccef53" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 } health_report { overall_health: UNKNOWN } } }
I20260810 15:16:09.526686 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.524384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.29.230.2:38535)
0810 15:16:09.524456 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.524458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.524465 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:09.524501 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.524502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.524522 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.524587 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:09.524723 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.525200 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.525930 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:09.526138 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:09.526192 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:09.526349 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:09.526472 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:09.526546 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.526583 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.526609 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:16:09.528668 6108 raft_consensus.cc:1261] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 1 FOLLOWER]: Refusing update from remote peer ecdf4663f1914d348fd87b51b6ccef53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:09.528692 5625 raft_consensus.cc:1261] T 90e470045dea4822ad48b7688dc44452 P 622a42adbe59493c8cd5d11627f49434 [term 1 FOLLOWER]: Refusing update from remote peer ecdf4663f1914d348fd87b51b6ccef53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:09.529062 6225 consensus_queue.cc:1035] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73e76306181f4ca7800af346ffaecfbb" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:09.529201 6217 consensus_queue.cc:1035] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "622a42adbe59493c8cd5d11627f49434" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:09.531430 6232 mvcc.cc:205] Tried to move back new op lower bound from 7316991875186388992 to 7316991875122520064. Current Snapshot: MvccSnapshot[applied={T|T < 7316991875186388992}]
I20260810 15:16:09.532750 6234 mvcc.cc:205] Tried to move back new op lower bound from 7316991875186388992 to 7316991875122520064. Current Snapshot: MvccSnapshot[applied={T|T < 7316991875186388992}]
I20260810 15:16:09.533032 6236 mvcc.cc:205] Tried to move back new op lower bound from 7316991875186388992 to 7316991875122520064. Current Snapshot: MvccSnapshot[applied={T|T < 7316991875186388992 or (T in {7316991875186388992})}]
I20260810 15:16:09.533919 5378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:09.534019 5378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:09.534772 5378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b88d91d393e430e8cd4eebf14e6446f: Sending DeleteTablet for 3 replicas of tablet 90e470045dea4822ad48b7688dc44452
I20260810 15:16:09.535074 5830 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e470045dea4822ad48b7688dc44452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:09 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:54442
I20260810 15:16:09.535125 5591 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e470045dea4822ad48b7688dc44452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:09 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:40228
I20260810 15:16:09.535125 6073 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e470045dea4822ad48b7688dc44452 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:09 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:46988
I20260810 15:16:09.535422 6253 tablet_replica.cc:330] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: stopping tablet replica
I20260810 15:16:09.535437 6263 tablet_replica.cc:330] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: stopping tablet replica
I20260810 15:16:09.535530 6263 raft_consensus.cc:2229] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:09.535566 6253 raft_consensus.cc:2229] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:09.535638 6263 raft_consensus.cc:2258] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:09.535701 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5489
I20260810 15:16:09.535737 6253 raft_consensus.cc:2258] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:09.544749 6263 ts_tablet_manager.cc:1905] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:09.546707 6263 ts_tablet_manager.cc:1918] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:09.546813 6263 log.cc:1196] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/90e470045dea4822ad48b7688dc44452
I20260810 15:16:09.547166 6263 ts_tablet_manager.cc:1939] T 90e470045dea4822ad48b7688dc44452 P 73e76306181f4ca7800af346ffaecfbb: Deleting consensus metadata
I20260810 15:16:09.547804 5350 catalog_manager.cc:4841] TS 73e76306181f4ca7800af346ffaecfbb (127.29.230.3:40679): tablet 90e470045dea4822ad48b7688dc44452 (table test-table [id=ee3623d5b11d4844971dfdbfe9c95d17]) successfully deleted
I20260810 15:16:09.551720 6253 ts_tablet_manager.cc:1905] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:09.553337 6253 ts_tablet_manager.cc:1918] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:09.553431 6253 log.cc:1196] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/90e470045dea4822ad48b7688dc44452
I20260810 15:16:09.553755 6253 ts_tablet_manager.cc:1939] T 90e470045dea4822ad48b7688dc44452 P ecdf4663f1914d348fd87b51b6ccef53: Deleting consensus metadata
W20260810 15:16:09.553937 5354 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:33909 (error 108)
I20260810 15:16:09.554221 5350 catalog_manager.cc:4841] TS ecdf4663f1914d348fd87b51b6ccef53 (127.29.230.2:38535): tablet 90e470045dea4822ad48b7688dc44452 (table test-table [id=ee3623d5b11d4844971dfdbfe9c95d17]) successfully deleted
W20260810 15:16:09.554611 6202 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:09.554343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:33909 (local address 127.0.0.1:40234)
0810 15:16:09.554391 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.554402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.554439 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.554562 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:33909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:16:09.554729 5354 catalog_manager.cc:4584] TS 622a42adbe59493c8cd5d11627f49434 (127.29.230.1:33909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90e470045dea4822ad48b7688dc44452: Network error: Client connection negotiation failed: client connection to 127.29.230.1:33909: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:16:09.554842 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5718
I20260810 15:16:09.564333 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5961
I20260810 15:16:09.573650 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5324
2026-08-10T15:16:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2184 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T15:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:09Z Disabled control of system clock
I20260810 15:16:09.611567 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36721
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:16:09.693187 6274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:09.693393 6274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:09.693414 6274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:09.694499 6274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:09.694533 6274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:09.694545 6274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:09.694556 6274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:09.694567 6274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:09.695936 6274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:09.696139 6274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:09.696384 6274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:09.701176 6289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.701218 6284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.701226 6274 server_base.cc:1034] running on GCE node
W20260810 15:16:09.701187 6283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.701675 6274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:09.701960 6274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:09.703104 6274 hybrid_clock.cc:648] HybridClock initialized: now 1786374969703079 us; error 43 us; skew 500 ppm
I20260810 15:16:09.704586 6274 webserver.cc:460] Webserver started at http://127.29.230.62:36303/ using document root <none> and password file <none>
I20260810 15:16:09.704777 6274 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:09.704816 6274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:09.704910 6274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:09.705971 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "ce270e8aa65ed52c3d4e1a44fc0ef733"
server_key_iv: "a5b107e8661268895bf9374fc6857b7e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.706265 6274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "ce270e8aa65ed52c3d4e1a44fc0ef733"
server_key_iv: "a5b107e8661268895bf9374fc6857b7e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.707481 6274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:09.708122 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.708273 6274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:09.708362 6274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "ce270e8aa65ed52c3d4e1a44fc0ef733"
server_key_iv: "a5b107e8661268895bf9374fc6857b7e"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.708418 6274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:09.711736 6274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:09.711957 6274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:09.712056 6274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:09.715765 6274 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36721
I20260810 15:16:09.715796 6395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36721 every 8 connection(s)
I20260810 15:16:09.716104 6274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:09.716778 6398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:09.719830 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: Bootstrap starting.
I20260810 15:16:09.720430 6398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:09.720714 6398 log.cc:824] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:09.721410 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: No bootstrap required, opened a new log
I20260810 15:16:09.723516 6398 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } }
I20260810 15:16:09.723678 6398 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:09.723701 6398 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7d21477a62f4f93b593f5d1c9dcb4f3, State: Initialized, Role: FOLLOWER
I20260810 15:16:09.723857 6398 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } }
I20260810 15:16:09.723924 6398 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:09.723946 6398 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:09.723974 6398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:09.724591 6398 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } }
I20260810 15:16:09.724687 6398 leader_election.cc:302] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7d21477a62f4f93b593f5d1c9dcb4f3; no voters:
I20260810 15:16:09.724870 6398 leader_election.cc:288] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:09.724920 6409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:09.725096 6409 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [term 1 LEADER]: Becoming Leader. State: Replica: e7d21477a62f4f93b593f5d1c9dcb4f3, State: Running, Role: LEADER
I20260810 15:16:09.725175 6398 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:09.725193 6409 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } }
I20260810 15:16:09.725618 6412 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } } }
I20260810 15:16:09.725736 6412 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:09.725842 6417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7d21477a62f4f93b593f5d1c9dcb4f3. Latest consensus state: current_term: 1 leader_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7d21477a62f4f93b593f5d1c9dcb4f3" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36721 } } }
I20260810 15:16:09.725924 6417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:09.725970 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 6274
I20260810 15:16:09.726068 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:09.726262 30616 external_mini_cluster.cc:1428] Setting key e40d24a08c74ff061764306ed624dd19
W20260810 15:16:09.727078 6432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:09.727245 6432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:09.727330 6424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:09.727504 6424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:09.728708 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.726993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.29.230.62:36721)
0810 15:16:09.727192 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.727204 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.727236 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:16:09.727381 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.727388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.727435 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.727577 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.728274 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.728277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.728278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:09.728281 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.728319 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:09.728331 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.728383 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.728460 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.728520 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260810 15:16:09.729169 6424 catalog_manager.cc:1347] Generated new cluster ID: 93fb396cac6e4c67bc89b0b6dc0f2641
I20260810 15:16:09.729223 6424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:09.736231 6424 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:09.736387 6424 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:09.743656 6424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: Generated new TSK 0
I20260810 15:16:09.743834 6424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:09.751510 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--builtin_ntp_servers=127.29.230.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:16:09.834057 6444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:09.834264 6444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:09.834285 6444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:09.835670 6444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:09.835726 6444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:09.835754 6444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:09.837235 6444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:09.837479 6444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:09.837781 6444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:09.842952 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.842940 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.842944 6451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.843303 6444 server_base.cc:1034] running on GCE node
I20260810 15:16:09.843528 6444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:09.843817 6444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:09.844981 6444 hybrid_clock.cc:648] HybridClock initialized: now 1786374969844961 us; error 38 us; skew 500 ppm
I20260810 15:16:09.846398 6444 webserver.cc:460] Webserver started at http://127.29.230.1:41813/ using document root <none> and password file <none>
I20260810 15:16:09.846796 6444 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:09.846889 6444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:09.847076 6444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:09.848526 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "1149df11f9114ff8a8d7f44deb4c5c2c"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c35b0b5f04a75b0404a8de2e486b6913"
server_key_iv: "ef5faa4b374a423614bc166c2fc57ed1"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.848979 6444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "1149df11f9114ff8a8d7f44deb4c5c2c"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c35b0b5f04a75b0404a8de2e486b6913"
server_key_iv: "ef5faa4b374a423614bc166c2fc57ed1"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.850602 6444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:09.851500 6465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.851680 6444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:09.851776 6444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "1149df11f9114ff8a8d7f44deb4c5c2c"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "c35b0b5f04a75b0404a8de2e486b6913"
server_key_iv: "ef5faa4b374a423614bc166c2fc57ed1"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.851846 6444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:09.855454 6444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:09.855683 6444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:09.855773 6444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:09.855933 6444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:09.856182 6444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:09.856211 6444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.856230 6444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:09.856248 6444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.861680 6444 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:34753
I20260810 15:16:09.861720 6673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:34753 every 8 connection(s)
I20260810 15:16:09.862056 6444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:09.865170 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.862815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:60217 (local address 127.29.230.62:36721)
0810 15:16:09.862936 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.862939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.863095 (+ 156us) server_negotiation.cc:407] Connection header received
0810 15:16:09.863677 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.863679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.863688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.863743 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.864922 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.864924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.864924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:09.864929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.864947 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:09.864951 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.865001 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.865080 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.865096 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:16:09.865363 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.862744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36721 (local address 127.29.230.1:60217)
0810 15:16:09.862970 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.863011 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.863573 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.863815 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.863819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.863829 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:09.864731 (+ 902us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:09.864732 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:09.864762 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:09.864792 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:09.865008 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:09.865014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:09.865017 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.865087 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.865152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260810 15:16:09.865917 6675 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36721
I20260810 15:16:09.866043 6675 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:09.866204 6675 heartbeater.cc:511] Master 127.29.230.62:36721 requested a full tablet report, sending...
I20260810 15:16:09.866719 6329 ts_manager.cc:194] Registered new tserver with Master: 1149df11f9114ff8a8d7f44deb4c5c2c (127.29.230.1:34753)
I20260810 15:16:09.867352 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 6444
I20260810 15:16:09.867446 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:09.867640 30616 external_mini_cluster.cc:1428] Setting key e97121752e8d712e2e82f40462414339
I20260810 15:16:09.868523 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--builtin_ntp_servers=127.29.230.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:16:09.950740 6678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:09.950950 6678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:09.950973 6678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:09.952147 6678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:09.952188 6678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:09.952212 6678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:09.953580 6678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:09.953814 6678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:09.954100 6678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:09.959005 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:09.959120 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.959156 6678 server_base.cc:1034] running on GCE node
W20260810 15:16:09.959018 6689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:09.959563 6678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:09.959767 6678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:09.960932 6678 hybrid_clock.cc:648] HybridClock initialized: now 1786374969960918 us; error 28 us; skew 500 ppm
I20260810 15:16:09.962244 6678 webserver.cc:460] Webserver started at http://127.29.230.2:42485/ using document root <none> and password file <none>
I20260810 15:16:09.962478 6678 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:09.962545 6678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:09.962679 6678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:09.963788 6678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "83fae2eb43c944e3b10f0091f29dcd65"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "faeedbd6bf278799642cdbdac31e67b5"
server_key_iv: "8fc7679b1241784b521863757cbc4a70"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.964110 6678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "83fae2eb43c944e3b10f0091f29dcd65"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "faeedbd6bf278799642cdbdac31e67b5"
server_key_iv: "8fc7679b1241784b521863757cbc4a70"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.965332 6678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:09.966154 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.966346 6678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:09.966439 6678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "83fae2eb43c944e3b10f0091f29dcd65"
format_stamp: "Formatted at 2026-08-10 15:16:09 on dist-test-slave-gjrz"
server_key: "faeedbd6bf278799642cdbdac31e67b5"
server_key_iv: "8fc7679b1241784b521863757cbc4a70"
server_key_version: "encryptionkey@0"
I20260810 15:16:09.966531 6678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:09.969812 6678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:09.970050 6678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:09.970168 6678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:09.970358 6678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:09.970667 6678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:09.970700 6678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.970728 6678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:09.970752 6678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:09.976128 6678 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42453
I20260810 15:16:09.976197 6907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42453 every 8 connection(s)
I20260810 15:16:09.976482 6678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:09.979501 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.977222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:41555 (local address 127.29.230.62:36721)
0810 15:16:09.977287 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:09.977290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:09.977491 (+ 201us) server_negotiation.cc:407] Connection header received
0810 15:16:09.977979 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:09.977981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:09.977989 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:09.978057 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:09.979212 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:09.979214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:09.979214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:09.979219 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:09.979237 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:09.979242 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:09.979332 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:09.979411 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 15:16:09.979429 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:16:09.979691 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:09.977189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36721 (local address 127.29.230.2:41555)
0810 15:16:09.977381 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:09.977414 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:09.977883 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:09.978135 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:09.978139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:09.978147 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:09.979031 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:09.979032 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:09.979075 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:09.979106 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:09.979338 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:09.979344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:09.979347 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:09.979417 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:16:09.979481 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260810 15:16:09.980192 6908 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36721
I20260810 15:16:09.980278 6908 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:09.980410 6908 heartbeater.cc:511] Master 127.29.230.62:36721 requested a full tablet report, sending...
I20260810 15:16:09.980751 6329 ts_manager.cc:194] Registered new tserver with Master: 83fae2eb43c944e3b10f0091f29dcd65 (127.29.230.2:42453)
I20260810 15:16:09.983861 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 6678
I20260810 15:16:09.983970 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:09.984133 30616 external_mini_cluster.cc:1428] Setting key d0c4f1fc950dadb34e06f1f0e9344d9f
I20260810 15:16:09.985002 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--builtin_ntp_servers=127.29.230.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 15:16:10.066753 6911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:10.066979 6911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:10.067003 6911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:10.068259 6911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:10.068302 6911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:10.068326 6911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:10.069756 6911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:10.070017 6911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:10.070338 6911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:10.075417 6920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.075564 6911 server_base.cc:1034] running on GCE node
W20260810 15:16:10.075481 6930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:10.075425 6927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.075812 6911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:10.076037 6911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:10.077193 6911 hybrid_clock.cc:648] HybridClock initialized: now 1786374970077179 us; error 33 us; skew 500 ppm
I20260810 15:16:10.078439 6911 webserver.cc:460] Webserver started at http://127.29.230.3:45145/ using document root <none> and password file <none>
I20260810 15:16:10.078688 6911 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:10.078734 6911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:10.078861 6911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:10.079980 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "df14a237cc1549299d9db830d77514f1"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "8cdade4ed2281f6fa2576df188f6e05d"
server_key_iv: "db56d23993591620757cdaa1860af980"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.080317 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "df14a237cc1549299d9db830d77514f1"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "8cdade4ed2281f6fa2576df188f6e05d"
server_key_iv: "db56d23993591620757cdaa1860af980"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.081643 6911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:10.082406 6942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.082576 6911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:10.082665 6911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "df14a237cc1549299d9db830d77514f1"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "8cdade4ed2281f6fa2576df188f6e05d"
server_key_iv: "db56d23993591620757cdaa1860af980"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.082728 6911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:10.086103 6911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:10.086359 6911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:10.086488 6911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:10.086679 6911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:10.087021 6911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:10.087092 6911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.087124 6911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:10.087152 6911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.093218 6911 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:38231
I20260810 15:16:10.093309 7140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:38231 every 8 connection(s)
I20260810 15:16:10.093650 6911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:10.096632 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.094447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:45449 (local address 127.29.230.62:36721)
0810 15:16:10.094510 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.094513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.094724 (+ 211us) server_negotiation.cc:407] Connection header received
0810 15:16:10.095172 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.095173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.095180 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.095243 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.096421 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.096423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.096423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.096428 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.096443 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.096448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.096499 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.096558 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.096572 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:10.096805 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.094381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36721 (local address 127.29.230.3:45449)
0810 15:16:10.094568 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.094610 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.095079 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.095399 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.095402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.095411 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.096271 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.096271 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.096301 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.096331 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.096503 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.096506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.096508 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.096559 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.096610 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260810 15:16:10.097289 7141 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36721
I20260810 15:16:10.097374 7141 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:10.097520 7141 heartbeater.cc:511] Master 127.29.230.62:36721 requested a full tablet report, sending...
I20260810 15:16:10.097874 6329 ts_manager.cc:194] Registered new tserver with Master: df14a237cc1549299d9db830d77514f1 (127.29.230.3:38231)
I20260810 15:16:10.100451 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 6911
I20260810 15:16:10.100541 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:10.100687 30616 external_mini_cluster.cc:1428] Setting key a6f0f464f8023545887d47dba2dcca77
I20260810 15:16:10.101630 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:16:10.103828 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.102870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38032 (local address 127.29.230.62:36721)
0810 15:16:10.102922 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.102924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.103016 (+ 92us) server_negotiation.cc:407] Connection header received
0810 15:16:10.103073 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.103074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.103079 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.103129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.103643 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.103645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.103645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.103649 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.103666 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.103670 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.103712 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.103767 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.103779 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:16:10.105335 6329 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:10.106787 6329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:10.110785 7160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.108814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:34753 (local address 127.0.0.1:57088)
0810 15:16:10.109167 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.109200 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.109257 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.109816 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.109817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.109820 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.110387 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.110388 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.110396 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.110411 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.110628 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.110629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.110629 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.110673 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.110697 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":100,"threads_started":1}
I20260810 15:16:10.110790 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.108920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.29.230.2:42453)
0810 15:16:10.109164 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.109169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.109302 (+ 133us) server_negotiation.cc:407] Connection header received
0810 15:16:10.109329 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.109332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.109344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.109410 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.110437 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.110447 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.110447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.110452 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.110489 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.110512 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.110584 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.110688 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.110703 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
I20260810 15:16:10.110942 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.109133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.29.230.3:38231)
0810 15:16:10.109309 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.109313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.109382 (+ 69us) server_negotiation.cc:407] Connection header received
0810 15:16:10.109434 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.109436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.109444 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.109511 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.110618 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.110625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.110625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.110629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.110658 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.110681 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.110734 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.110850 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.110865 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260810 15:16:10.111332 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.108925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.29.230.1:34753)
0810 15:16:10.109250 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.109254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.109265 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:10.109335 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.109344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.109354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.109421 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.110478 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.110486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.110486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.110491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.110525 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.110543 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.110603 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.111219 (+ 616us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.111234 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":92,"threads_started":1}
I20260810 15:16:10.110821 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.108804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42453 (local address 127.0.0.1:35186)
0810 15:16:10.109140 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.109182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.109248 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.109823 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.109824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.109826 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.110312 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.110312 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.110330 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.110355 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.110589 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.110596 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.110598 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.110670 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.110704 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":18,"thread_start_us":74,"threads_started":1}
I20260810 15:16:10.110909 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.109030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:38231 (local address 127.0.0.1:47466)
0810 15:16:10.109323 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.109332 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.109381 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.109522 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.109529 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.109537 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.110340 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.110341 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.110345 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.110356 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.110760 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.110761 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.110762 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.110803 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.110815 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":82,"threads_started":1}
I20260810 15:16:10.112100 6558 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 (DEFAULT_TABLE table=test-table [id=80e502122fb8496bb75b7b2676c24bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:10.112224 6802 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 (DEFAULT_TABLE table=test-table [id=80e502122fb8496bb75b7b2676c24bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:10.112459 6558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e1c5b6d54419b18503f8988fde96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.112581 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e1c5b6d54419b18503f8988fde96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.115077 7172 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: Bootstrap starting.
I20260810 15:16:10.115098 7174 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Bootstrap starting.
I20260810 15:16:10.115765 7174 tablet_bootstrap.cc:654] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.115787 7172 tablet_bootstrap.cc:654] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.116043 7174 log.cc:824] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.116005 7026 tablet_service.cc:1467] Processing CreateTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 (DEFAULT_TABLE table=test-table [id=80e502122fb8496bb75b7b2676c24bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:10.116232 7172 log.cc:824] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.116364 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e82e1c5b6d54419b18503f8988fde96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.116698 7174 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: No bootstrap required, opened a new log
I20260810 15:16:10.116760 7174 ts_tablet_manager.cc:1397] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:10.118633 7174 raft_consensus.cc:348] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.118770 7174 raft_consensus.cc:374] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.118768 7178 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Bootstrap starting.
I20260810 15:16:10.118803 7174 raft_consensus.cc:729] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fae2eb43c944e3b10f0091f29dcd65, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.118937 7174 consensus_queue.cc:260] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.119184 7174 ts_tablet_manager.cc:1428] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:16:10.119328 6908 heartbeater.cc:503] Master 127.29.230.62:36721 was elected leader, sending a full tablet report...
I20260810 15:16:10.119412 7178 tablet_bootstrap.cc:654] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.119444 7172 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: No bootstrap required, opened a new log
I20260810 15:16:10.119500 7172 ts_tablet_manager.cc:1397] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:16:10.119724 7178 log.cc:824] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.120491 7178 tablet_bootstrap.cc:492] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: No bootstrap required, opened a new log
I20260810 15:16:10.120556 7178 ts_tablet_manager.cc:1397] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:10.121934 7172 raft_consensus.cc:348] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.122102 7172 raft_consensus.cc:374] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.122139 7172 raft_consensus.cc:729] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1149df11f9114ff8a8d7f44deb4c5c2c, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.122259 7172 consensus_queue.cc:260] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.122534 7172 ts_tablet_manager.cc:1428] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:16:10.122552 6675 heartbeater.cc:503] Master 127.29.230.62:36721 was elected leader, sending a full tablet report...
I20260810 15:16:10.122967 7178 raft_consensus.cc:348] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.123133 7178 raft_consensus.cc:374] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.123183 7178 raft_consensus.cc:729] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df14a237cc1549299d9db830d77514f1, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.123311 7178 consensus_queue.cc:260] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.123584 7178 ts_tablet_manager.cc:1428] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 15:16:10.123605 7141 heartbeater.cc:503] Master 127.29.230.62:36721 was elected leader, sending a full tablet report...
W20260810 15:16:10.227051 6909 tablet.cc:2369] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:10.229921 7184 raft_consensus.cc:482] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:10.230057 7184 raft_consensus.cc:504] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.230376 7184 leader_election.cc:288] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83fae2eb43c944e3b10f0091f29dcd65 (127.29.230.2:42453), 1149df11f9114ff8a8d7f44deb4c5c2c (127.29.230.1:34753)
I20260810 15:16:10.231916 7144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.230604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42453 (local address 127.29.230.3:40937)
0810 15:16:10.230682 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.230695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.230755 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.230960 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.230960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.230963 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.231557 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.231557 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.231566 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.231580 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.231784 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.231785 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.231786 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.231822 (+ 36us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.231837 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 15:16:10.231926 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.230676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:40937 (local address 127.29.230.2:42453)
0810 15:16:10.230745 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.230748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.230761 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:16:10.230814 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.230815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.230822 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.230883 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.231682 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.231684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.231684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.231690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.231709 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.231715 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.231781 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.231834 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.231849 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:10.231985 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.230686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:50553 (local address 127.29.230.1:34753)
0810 15:16:10.230733 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.230736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.230936 (+ 200us) server_negotiation.cc:407] Connection header received
0810 15:16:10.230972 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.230973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.230978 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.231027 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.231768 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.231770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.231770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.231774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.231791 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.231795 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.231852 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.231900 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.231917 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:10.232040 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.230604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:34753 (local address 127.29.230.3:50553)
0810 15:16:10.230862 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.230870 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.230917 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.231042 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.231043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.231045 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:10.231678 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:10.231678 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:10.231684 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:10.231694 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:10.231855 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:10.231855 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:10.231856 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:16:10.231900 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:10.231915 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":106,"thread_start_us":102,"threads_started":1}
I20260810 15:16:10.232419 6593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e1c5b6d54419b18503f8988fde96" candidate_uuid: "df14a237cc1549299d9db830d77514f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" is_pre_election: true
I20260810 15:16:10.232600 6593 raft_consensus.cc:2454] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df14a237cc1549299d9db830d77514f1 in term 0.
I20260810 15:16:10.232591 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e1c5b6d54419b18503f8988fde96" candidate_uuid: "df14a237cc1549299d9db830d77514f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fae2eb43c944e3b10f0091f29dcd65" is_pre_election: true
I20260810 15:16:10.232738 6835 raft_consensus.cc:2454] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df14a237cc1549299d9db830d77514f1 in term 0.
I20260810 15:16:10.232884 6944 leader_election.cc:302] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1149df11f9114ff8a8d7f44deb4c5c2c, df14a237cc1549299d9db830d77514f1; no voters:
I20260810 15:16:10.233083 7184 raft_consensus.cc:2781] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:10.233156 7184 raft_consensus.cc:482] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:10.233182 7184 raft_consensus.cc:3037] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.233899 7184 raft_consensus.cc:504] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.234021 7184 leader_election.cc:288] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [CANDIDATE]: Term 1 election: Requested vote from peers 83fae2eb43c944e3b10f0091f29dcd65 (127.29.230.2:42453), 1149df11f9114ff8a8d7f44deb4c5c2c (127.29.230.1:34753)
I20260810 15:16:10.234202 6835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e1c5b6d54419b18503f8988fde96" candidate_uuid: "df14a237cc1549299d9db830d77514f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fae2eb43c944e3b10f0091f29dcd65"
I20260810 15:16:10.234278 6835 raft_consensus.cc:3037] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.235013 6835 raft_consensus.cc:2454] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df14a237cc1549299d9db830d77514f1 in term 1.
I20260810 15:16:10.235134 6593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e82e1c5b6d54419b18503f8988fde96" candidate_uuid: "df14a237cc1549299d9db830d77514f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c"
I20260810 15:16:10.235219 6593 raft_consensus.cc:3037] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.235311 6945 leader_election.cc:302] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fae2eb43c944e3b10f0091f29dcd65, df14a237cc1549299d9db830d77514f1; no voters:
I20260810 15:16:10.235392 7184 raft_consensus.cc:2781] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:10.235529 7184 raft_consensus.cc:686] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 1 LEADER]: Becoming Leader. State: Replica: df14a237cc1549299d9db830d77514f1, State: Running, Role: LEADER
I20260810 15:16:10.235630 7184 consensus_queue.cc:237] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } }
I20260810 15:16:10.236280 6593 raft_consensus.cc:2454] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df14a237cc1549299d9db830d77514f1 in term 1.
I20260810 15:16:10.236512 6329 catalog_manager.cc:5495] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to df14a237cc1549299d9db830d77514f1 (127.29.230.3). New cstate: current_term: 1 leader_uuid: "df14a237cc1549299d9db830d77514f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df14a237cc1549299d9db830d77514f1" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 } health_report { overall_health: UNKNOWN } } }
I20260810 15:16:10.256425 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.255472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.29.230.3:38231)
0810 15:16:10.255539 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.255541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.255551 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:10.255615 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.255616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.255635 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.255681 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.256206 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.256207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.256207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.256211 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.256227 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.256231 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.256280 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.256333 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.256346 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:10.258275 6593 raft_consensus.cc:1261] T 9e82e1c5b6d54419b18503f8988fde96 P 1149df11f9114ff8a8d7f44deb4c5c2c [term 1 FOLLOWER]: Refusing update from remote peer df14a237cc1549299d9db830d77514f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:10.258612 6835 raft_consensus.cc:1261] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 1 FOLLOWER]: Refusing update from remote peer df14a237cc1549299d9db830d77514f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:10.258641 7184 consensus_queue.cc:1035] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1149df11f9114ff8a8d7f44deb4c5c2c" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 34753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:10.258930 7184 consensus_queue.cc:1035] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fae2eb43c944e3b10f0091f29dcd65" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:10.261072 7196 mvcc.cc:205] Tried to move back new op lower bound from 7316991878175387648 to 7316991878085840896. Current Snapshot: MvccSnapshot[applied={T|T < 7316991878175387648}]
I20260810 15:16:10.261242 7203 mvcc.cc:205] Tried to move back new op lower bound from 7316991878175387648 to 7316991878085840896. Current Snapshot: MvccSnapshot[applied={T|T < 7316991878175387648}]
I20260810 15:16:10.264097 6329 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:10.264187 6329 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:10.264889 6329 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7d21477a62f4f93b593f5d1c9dcb4f3: Sending DeleteTablet for 3 replicas of tablet 9e82e1c5b6d54419b18503f8988fde96
I20260810 15:16:10.265229 6802 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:10 UTC) from {username='slave'} at 127.0.0.1:35186
I20260810 15:16:10.265245 7026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:10 UTC) from {username='slave'} at 127.0.0.1:47466
I20260810 15:16:10.265458 7212 tablet_replica.cc:330] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: stopping tablet replica
I20260810 15:16:10.265561 7212 raft_consensus.cc:2229] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:10.265691 7212 raft_consensus.cc:2258] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:10.265735 7214 tablet_replica.cc:330] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: stopping tablet replica
I20260810 15:16:10.265724 6558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e82e1c5b6d54419b18503f8988fde96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:10 UTC) from {username='slave'} at 127.0.0.1:57088
I20260810 15:16:10.265781 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 6444
I20260810 15:16:10.265834 7214 raft_consensus.cc:2229] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:10.265916 7214 raft_consensus.cc:2258] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:10.275478 7212 ts_tablet_manager.cc:1905] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:10.277637 7212 ts_tablet_manager.cc:1918] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:10.277731 7212 log.cc:1196] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/9e82e1c5b6d54419b18503f8988fde96
I20260810 15:16:10.278069 7212 ts_tablet_manager.cc:1939] T 9e82e1c5b6d54419b18503f8988fde96 P df14a237cc1549299d9db830d77514f1: Deleting consensus metadata
I20260810 15:16:10.297217 7214 ts_tablet_manager.cc:1905] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:10.299007 7214 ts_tablet_manager.cc:1918] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:10.299103 7214 log.cc:1196] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/9e82e1c5b6d54419b18503f8988fde96
I20260810 15:16:10.299381 7214 ts_tablet_manager.cc:1939] T 9e82e1c5b6d54419b18503f8988fde96 P 83fae2eb43c944e3b10f0091f29dcd65: Deleting consensus metadata
I20260810 15:16:10.299724 6307 catalog_manager.cc:4841] TS df14a237cc1549299d9db830d77514f1 (127.29.230.3:38231): tablet 9e82e1c5b6d54419b18503f8988fde96 (table test-table [id=80e502122fb8496bb75b7b2676c24bb3]) successfully deleted
I20260810 15:16:10.299867 6307 catalog_manager.cc:4841] TS 83fae2eb43c944e3b10f0091f29dcd65 (127.29.230.2:42453): tablet 9e82e1c5b6d54419b18503f8988fde96 (table test-table [id=80e502122fb8496bb75b7b2676c24bb3]) successfully deleted
W20260810 15:16:10.300647 6306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:34753 (error 108)
I20260810 15:16:10.300895 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 6678
I20260810 15:16:10.301441 7169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.301037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:34753 (local address 127.0.0.1:57100)
0810 15:16:10.301366 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.301407 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:34753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
W20260810 15:16:10.303478 6306 catalog_manager.cc:4584] TS 1149df11f9114ff8a8d7f44deb4c5c2c (127.29.230.1:34753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e82e1c5b6d54419b18503f8988fde96: Network error: Client connection negotiation failed: client connection to 127.29.230.1:34753: connect: Connection refused (error 111)
I20260810 15:16:10.318362 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 6911
I20260810 15:16:10.346848 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 6274
2026-08-10T15:16:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (766 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 15:16:10.370826 30616 mini_oidc.cc:152] Starting JWKS server
I20260810 15:16:10.372104 30616 webserver.cc:460] Webserver started at https://127.0.0.1:33705/ using document root <none> and password file <none>
I20260810 15:16:10.372186 30616 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:16:10.372320 30616 webserver.cc:460] Webserver started at http://127.0.0.1:43757/ using document root <none> and password file <none>
2026-08-10T15:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:10Z Disabled control of system clock
I20260810 15:16:10.381237 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37845
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:37845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/testchainca.cert with env {}
W20260810 15:16:10.463389 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:10.463616 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:10.463640 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:10.465034 7223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:10.465094 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:10.465109 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:10.465121 7223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:10.465132 7223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:16:10.465147 7223 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:16:10.465173 7223 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33705/jwks.json
I20260810 15:16:10.466966 7223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:37845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:10.467229 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:10.467476 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:10.472256 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:10.472270 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:10.472270 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.472580 7223 server_base.cc:1034] running on GCE node
I20260810 15:16:10.472843 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:10.473194 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:10.474380 7223 hybrid_clock.cc:648] HybridClock initialized: now 1786374970474361 us; error 30 us; skew 500 ppm
I20260810 15:16:10.475754 7223 webserver.cc:460] Webserver started at http://127.29.230.62:40901/ using document root <none> and password file <none>
I20260810 15:16:10.476007 7223 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:10.476063 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:10.476212 7223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:10.477341 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "b466fc4060664524a71b3e5e19bf1be0"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "e0c63aa93e8284fa64ced65603488294"
server_key_iv: "4f890f64c5560cf3786486811cdfbfe8"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.477682 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "b466fc4060664524a71b3e5e19bf1be0"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "e0c63aa93e8284fa64ced65603488294"
server_key_iv: "4f890f64c5560cf3786486811cdfbfe8"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.478900 7223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:10.479671 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.479825 7223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.479920 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "b466fc4060664524a71b3e5e19bf1be0"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "e0c63aa93e8284fa64ced65603488294"
server_key_iv: "4f890f64c5560cf3786486811cdfbfe8"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.479984 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:10.505409 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:10.505741 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:10.505882 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:10.509856 7223 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:37845
I20260810 15:16:10.509886 7342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:37845 every 8 connection(s)
I20260810 15:16:10.510241 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:10.511327 7343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.513700 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0: Bootstrap starting.
I20260810 15:16:10.514304 7343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.514590 7343 log.cc:824] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.515338 7343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0: No bootstrap required, opened a new log
I20260810 15:16:10.515808 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 7223
I20260810 15:16:10.515950 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:10.516116 30616 external_mini_cluster.cc:1428] Setting key caec108314a8aed04ee4fc7c2962a8be
I20260810 15:16:10.517378 7343 raft_consensus.cc:348] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } }
I20260810 15:16:10.517566 7343 raft_consensus.cc:374] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.517608 7343 raft_consensus.cc:729] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b466fc4060664524a71b3e5e19bf1be0, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.517741 7343 consensus_queue.cc:260] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } }
I20260810 15:16:10.517815 7343 raft_consensus.cc:388] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:10.517838 7343 raft_consensus.cc:482] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:10.517882 7343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.518407 7343 raft_consensus.cc:504] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } }
I20260810 15:16:10.518501 7343 leader_election.cc:302] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b466fc4060664524a71b3e5e19bf1be0; no voters:
I20260810 15:16:10.518664 7343 leader_election.cc:288] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:10.518711 7349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:10.518827 7349 raft_consensus.cc:686] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [term 1 LEADER]: Becoming Leader. State: Replica: b466fc4060664524a71b3e5e19bf1be0, State: Running, Role: LEADER
I20260810 15:16:10.518905 7349 consensus_queue.cc:237] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } }
I20260810 15:16:10.518929 7343 sys_catalog.cc:564] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:10.519249 7349 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b466fc4060664524a71b3e5e19bf1be0. Latest consensus state: current_term: 1 leader_uuid: "b466fc4060664524a71b3e5e19bf1be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } } }
I20260810 15:16:10.519348 7349 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.519595 7359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:10.520294 7359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:10.520335 7355 sys_catalog.cc:455] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b466fc4060664524a71b3e5e19bf1be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b466fc4060664524a71b3e5e19bf1be0" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37845 } } }
I20260810 15:16:10.520416 7355 sys_catalog.cc:458] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.521976 7386 mvcc.cc:205] Tried to move back new op lower bound from 7316991879254913024 to 7316991879245869056. Current Snapshot: MvccSnapshot[applied={T|T < 7316991879254913024}]
I20260810 15:16:10.522482 7359 catalog_manager.cc:1347] Generated new cluster ID: 58b0716751814c51aa20a2aa668bb2dd
I20260810 15:16:10.522500 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.516617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.29.230.62:37845)
0810 15:16:10.516816 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.516824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.516849 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:16:10.517314 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.517321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.517380 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.517499 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.517743 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.519099 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.520134 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.520756 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.520872 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.521600 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.521604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.521604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.521815 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.521884 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.521920 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.521988 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.522201 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.522283 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260810 15:16:10.522541 7359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:10.523456 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:16:10.531827 7359 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:10.532580 7359 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:10.540510 7359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b466fc4060664524a71b3e5e19bf1be0: Generated new TSK 0
I20260810 15:16:10.540714 7359 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 15:16:10.573055 7396 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 15:16:10.573184 7396 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 15:16:10.573436 7348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.557320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.29.230.62:37845)
0810 15:16:10.557371 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.557373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.557498 (+ 125us) server_negotiation.cc:407] Connection header received
0810 15:16:10.557584 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.557585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.557598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.557680 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:16:10.557806 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.558607 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.559443 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.559754 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.559825 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.560035 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:16:10.573184 (+ 13149us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 15:16:10.573356 (+ 172us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260810 15:16:10.573503 7348 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35072: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 15:16:10.574124 7395 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:10.557209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37845 (local address 127.0.0.1:35072)
0810 15:16:10.557424 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.557436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.557500 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.557684 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.557684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.557686 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:16:10.557752 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:10.557754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.558693 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.558694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:10.559370 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:10.559373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.559836 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.559836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:10.559878 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.559900 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:16:10.573395 (+ 13495us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 15:16:10.573781 (+ 386us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.230.62:37845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.230.62:37845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 15:16:10.574721 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 7223
2026-08-10T15:16:10Z chronyd exiting
I20260810 15:16:10.622534 30616 test_util.cc:175] -----------------------------------------------
I20260810 15:16:10.622614 30616 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786374947162751-30616-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (252 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 15:16:10.623396 30616 mini_oidc.cc:152] Starting JWKS server
I20260810 15:16:10.624101 30616 webserver.cc:460] Webserver started at https://127.0.0.1:38995/ using document root <none> and password file <none>
I20260810 15:16:10.624143 30616 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:16:10.624280 30616 webserver.cc:460] Webserver started at http://127.0.0.1:45465/ using document root <none> and password file <none>
2026-08-10T15:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:10Z Disabled control of system clock
I20260810 15:16:10.633103 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35119
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:35119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38995/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/testchainca.cert with env {}
W20260810 15:16:10.713881 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:10.714088 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:10.714109 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:10.715282 7408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:10.715322 7408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:10.715334 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:10.715356 7408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:10.715379 7408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:16:10.715404 7408 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:16:10.715423 7408 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38995/jwks.json
I20260810 15:16:10.716943 7408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:35119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38995/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:10.717170 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:10.717368 7408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:10.722054 7418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:10.722064 7414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.722152 7408 server_base.cc:1034] running on GCE node
W20260810 15:16:10.722064 7421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.722564 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:10.722843 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:10.724036 7408 hybrid_clock.cc:648] HybridClock initialized: now 1786374970724006 us; error 45 us; skew 500 ppm
I20260810 15:16:10.725337 7408 webserver.cc:460] Webserver started at http://127.29.230.62:35227/ using document root <none> and password file <none>
I20260810 15:16:10.725579 7408 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:10.725625 7408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:10.725746 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:10.726827 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "f011ad82fb774c44a094120040f58542"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "dae37236d93f41f5a2a010402cdcf35d"
server_key_iv: "9e354a50e8f204deda77df97f9dd6e7d"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.727182 7408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "f011ad82fb774c44a094120040f58542"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "dae37236d93f41f5a2a010402cdcf35d"
server_key_iv: "9e354a50e8f204deda77df97f9dd6e7d"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.728436 7408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:10.729197 7431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.729382 7408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:10.729490 7408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "f011ad82fb774c44a094120040f58542"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "dae37236d93f41f5a2a010402cdcf35d"
server_key_iv: "9e354a50e8f204deda77df97f9dd6e7d"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.729557 7408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:10.743040 7408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:10.743376 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:10.743510 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:10.747500 7408 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:35119
I20260810 15:16:10.747543 7534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:35119 every 8 connection(s)
I20260810 15:16:10.747855 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:10.748543 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.751312 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542: Bootstrap starting.
I20260810 15:16:10.751904 7535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.752164 7535 log.cc:824] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.752945 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542: No bootstrap required, opened a new log
I20260810 15:16:10.754621 7535 raft_consensus.cc:348] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } }
I20260810 15:16:10.754765 7535 raft_consensus.cc:374] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.754787 7535 raft_consensus.cc:729] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f011ad82fb774c44a094120040f58542, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.754918 7535 consensus_queue.cc:260] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } }
I20260810 15:16:10.754983 7535 raft_consensus.cc:388] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:10.755004 7535 raft_consensus.cc:482] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:10.755031 7535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.755630 7535 raft_consensus.cc:504] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } }
I20260810 15:16:10.755724 7535 leader_election.cc:302] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f011ad82fb774c44a094120040f58542; no voters:
I20260810 15:16:10.755892 7535 leader_election.cc:288] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:10.755941 7540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:10.756102 7540 raft_consensus.cc:686] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [term 1 LEADER]: Becoming Leader. State: Replica: f011ad82fb774c44a094120040f58542, State: Running, Role: LEADER
I20260810 15:16:10.756167 7535 sys_catalog.cc:564] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:10.756215 7540 consensus_queue.cc:237] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } }
I20260810 15:16:10.756634 7541 sys_catalog.cc:455] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f011ad82fb774c44a094120040f58542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } } }
I20260810 15:16:10.756727 7541 sys_catalog.cc:458] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.756863 7542 sys_catalog.cc:455] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f011ad82fb774c44a094120040f58542. Latest consensus state: current_term: 1 leader_uuid: "f011ad82fb774c44a094120040f58542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f011ad82fb774c44a094120040f58542" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35119 } } }
I20260810 15:16:10.756944 7559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:10.757473 7559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:10.757557 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 7408
I20260810 15:16:10.757673 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:10.757855 30616 external_mini_cluster.cc:1428] Setting key f0c9581cf3156bdf888a3a6a06f6d977
I20260810 15:16:10.756947 7542 sys_catalog.cc:458] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.759238 7559 catalog_manager.cc:1347] Generated new cluster ID: 1eb677d67a104f4889eb70b860af30c0
I20260810 15:16:10.759318 7559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:10.763837 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.758278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33804 (local address 127.29.230.62:35119)
0810 15:16:10.758476 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.758485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.759563 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 15:16:10.759683 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.759689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.759744 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.759860 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.760058 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.760946 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.761710 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.762261 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.762379 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.763127 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.763131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.763132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.763339 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.763376 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.763394 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.763452 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.763557 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.763644 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260810 15:16:10.764891 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:16:10.769943 7559 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:10.770615 7559 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:10.777872 7559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f011ad82fb774c44a094120040f58542: Generated new TSK 0
I20260810 15:16:10.778077 7559 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 15:16:10.802086 7602 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 15:16:10.802181 7602 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 15:16:10.802506 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.786399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.29.230.62:35119)
0810 15:16:10.786447 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.786449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.786558 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:16:10.786635 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.786636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.786648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.786709 (+ 61us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:16:10.786870 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.787437 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.788211 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.788429 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.788493 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.788660 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:16:10.802262 (+ 13602us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 15:16:10.802428 (+ 166us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 15:16:10.802568 7601 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:10.786341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35119 (local address 127.0.0.1:33814)
0810 15:16:10.786495 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:10.786506 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:10.786572 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:10.786723 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:10.786724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:10.786726 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:16:10.786812 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:10.786814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.787512 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.787514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:10.788132 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:10.788135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.788496 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.788497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:10.788565 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.788571 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:16:10.802382 (+ 13811us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 15:16:10.802479 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.230.62:35119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":105,"threads_started":1}
W20260810 15:16:10.802570 7590 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33814: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38995/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260810 15:16:10.802863 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 7408
2026-08-10T15:16:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 15:16:10.856741 30616 mini_oidc.cc:152] Starting JWKS server
I20260810 15:16:10.857419 30616 webserver.cc:460] Webserver started at https://127.0.0.1:35263/ using document root <none> and password file <none>
I20260810 15:16:10.857462 30616 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 15:16:10.857580 30616 webserver.cc:460] Webserver started at http://127.0.0.1:45649/ using document root <none> and password file <none>
2026-08-10T15:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:10Z Disabled control of system clock
I20260810 15:16:10.865932 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34239
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34239
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35263/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 15:16:10.946460 7610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:10.946658 7610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:10.946688 7610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:10.947866 7610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:10.947916 7610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:10.947934 7610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:10.947949 7610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:10.947965 7610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:16:10.947986 7610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:16:10.948005 7610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35263/jwks.json
I20260810 15:16:10.949381 7610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35263/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:10.949615 7610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:10.949899 7610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:10.954705 7616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.954885 7610 server_base.cc:1034] running on GCE node
W20260810 15:16:10.954700 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:10.954700 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:10.955192 7610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:10.955516 7610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:10.956692 7610 hybrid_clock.cc:648] HybridClock initialized: now 1786374970956673 us; error 35 us; skew 500 ppm
I20260810 15:16:10.957962 7610 webserver.cc:460] Webserver started at http://127.29.230.62:33139/ using document root <none> and password file <none>
I20260810 15:16:10.958257 7610 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:10.958307 7610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:10.958431 7610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:10.959560 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "e99d9adc5c71469cb3ffab354ee9a094"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "240b0f59df7d0c53ebedf7b50a9efa15"
server_key_iv: "2a913643593b1cad29ca3144092cb6be"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.959906 7610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "e99d9adc5c71469cb3ffab354ee9a094"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "240b0f59df7d0c53ebedf7b50a9efa15"
server_key_iv: "2a913643593b1cad29ca3144092cb6be"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.961172 7610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:10.961867 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:10.962112 7610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:10.962206 7610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e99d9adc5c71469cb3ffab354ee9a094"
format_stamp: "Formatted at 2026-08-10 15:16:10 on dist-test-slave-gjrz"
server_key: "240b0f59df7d0c53ebedf7b50a9efa15"
server_key_iv: "2a913643593b1cad29ca3144092cb6be"
server_key_version: "encryptionkey@0"
I20260810 15:16:10.962275 7610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:10.975798 7610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:10.976110 7610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:10.976239 7610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:10.980273 7610 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34239
I20260810 15:16:10.980307 7734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34239 every 8 connection(s)
I20260810 15:16:10.980633 7610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:10.981294 7739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:10.983994 7739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094: Bootstrap starting.
I20260810 15:16:10.984555 7739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:10.984824 7739 log.cc:824] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:10.985594 7739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094: No bootstrap required, opened a new log
I20260810 15:16:10.987192 7739 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } }
I20260810 15:16:10.987366 7739 raft_consensus.cc:374] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:10.987399 7739 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99d9adc5c71469cb3ffab354ee9a094, State: Initialized, Role: FOLLOWER
I20260810 15:16:10.987538 7739 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } }
I20260810 15:16:10.987601 7739 raft_consensus.cc:388] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:10.987622 7739 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:10.987648 7739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:10.988165 7739 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } }
I20260810 15:16:10.988256 7739 leader_election.cc:302] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e99d9adc5c71469cb3ffab354ee9a094; no voters:
I20260810 15:16:10.988416 7739 leader_election.cc:288] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:10.988467 7742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:10.988612 7742 raft_consensus.cc:686] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [term 1 LEADER]: Becoming Leader. State: Replica: e99d9adc5c71469cb3ffab354ee9a094, State: Running, Role: LEADER
I20260810 15:16:10.988675 7739 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:10.988715 7742 consensus_queue.cc:237] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } }
I20260810 15:16:10.989102 7743 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e99d9adc5c71469cb3ffab354ee9a094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } } }
I20260810 15:16:10.989217 7743 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.989298 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99d9adc5c71469cb3ffab354ee9a094. Latest consensus state: current_term: 1 leader_uuid: "e99d9adc5c71469cb3ffab354ee9a094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e99d9adc5c71469cb3ffab354ee9a094" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34239 } } }
I20260810 15:16:10.989367 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:10.989468 7758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:10.989974 7758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:10.990260 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 7610
I20260810 15:16:10.990362 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:10.990557 30616 external_mini_cluster.cc:1428] Setting key 0e212573f5572679c1c7dd9f20b4d03f
I20260810 15:16:10.991709 7758 catalog_manager.cc:1347] Generated new cluster ID: 60cd2717ce994c518d446bd20e42d626
I20260810 15:16:10.991760 7758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:10.996636 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:10.991016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.29.230.62:34239)
0810 15:16:10.991330 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:10.991339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:10.991372 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:16:10.991487 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:10.991494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:10.991562 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:10.991976 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:10.992070 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.993453 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.994508 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:10.995162 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:10.995314 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:10.995951 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:10.995955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:10.995955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:10.996159 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:10.996203 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:10.996218 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:10.996291 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:10.996356 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 15:16:10.996439 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":65,"threads_started":1}
I20260810 15:16:10.997784 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:16:11.016180 7758 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:11.016710 7758 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:11.020459 7758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e99d9adc5c71469cb3ffab354ee9a094: Generated new TSK 0
I20260810 15:16:11.020639 7758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:11.028607 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:11.024830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.29.230.62:34239)
0810 15:16:11.024869 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:11.024871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:11.025007 (+ 136us) server_negotiation.cc:407] Connection header received
0810 15:16:11.025082 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:11.025083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:11.025096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:11.025159 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:16:11.025276 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.026068 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.026799 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.027133 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.027202 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:11.027458 (+ 256us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:16:11.028404 (+ 946us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 15:16:11.028540 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:16:11.028664 7775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53628: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 15:16:11.030457 7792 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:11.024781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34239 (local address 127.0.0.1:53628)
0810 15:16:11.024929 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:11.024941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:11.025009 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:11.025168 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:11.025169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:11.025171 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:16:11.025226 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:11.025229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.026138 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.026138 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:11.026729 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:11.026731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.027248 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.027250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:11.027356 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:11.027362 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:16:11.028525 (+ 1163us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 15:16:11.030352 (+ 1827us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.230.62:34239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35263/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":1637,"thread_start_us":84,"threads_started":1}
I20260810 15:16:11.030719 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 7610
E20260810 15:16:11.054949 7794 webserver.cc:533] Webserver: sslize failed:
E20260810 15:16:11.055032 7794 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T15:16:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T15:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:11Z Disabled control of system clock
I20260810 15:16:11.098265 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36207
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36207
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/testchainca.cert with env {}
W20260810 15:16:11.180691 7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:11.180907 7798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:11.180927 7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:11.182122 7798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:11.182165 7798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:11.182178 7798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:11.182188 7798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:11.182199 7798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 15:16:11.182209 7798 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 15:16:11.182219 7798 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 15:16:11.183736 7798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36207
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:11.183946 7798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:11.184109 7798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:11.189008 7807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:11.189030 7808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:11.189076 7798 server_base.cc:1034] running on GCE node
W20260810 15:16:11.189325 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:11.189641 7798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:11.189924 7798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:11.191095 7798 hybrid_clock.cc:648] HybridClock initialized: now 1786374971191074 us; error 37 us; skew 500 ppm
I20260810 15:16:11.192516 7798 webserver.cc:460] Webserver started at http://127.29.230.62:36345/ using document root <none> and password file <none>
I20260810 15:16:11.192721 7798 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:11.192760 7798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:11.192853 7798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:11.193897 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "35075730391b4ea48e19695413cf28ab"
format_stamp: "Formatted at 2026-08-10 15:16:11 on dist-test-slave-gjrz"
server_key: "3a572ea6b64dc8a011c7966681386226"
server_key_iv: "ab8ec5f17ddff23c37b377b16db90716"
server_key_version: "encryptionkey@0"
I20260810 15:16:11.194203 7798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "35075730391b4ea48e19695413cf28ab"
format_stamp: "Formatted at 2026-08-10 15:16:11 on dist-test-slave-gjrz"
server_key: "3a572ea6b64dc8a011c7966681386226"
server_key_iv: "ab8ec5f17ddff23c37b377b16db90716"
server_key_version: "encryptionkey@0"
I20260810 15:16:11.195519 7798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:11.196208 7824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:11.196359 7798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:11.196439 7798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "35075730391b4ea48e19695413cf28ab"
format_stamp: "Formatted at 2026-08-10 15:16:11 on dist-test-slave-gjrz"
server_key: "3a572ea6b64dc8a011c7966681386226"
server_key_iv: "ab8ec5f17ddff23c37b377b16db90716"
server_key_version: "encryptionkey@0"
I20260810 15:16:11.196513 7798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:11.207093 7798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:11.207404 7798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:11.207532 7798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:11.211371 7798 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36207
I20260810 15:16:11.211423 7911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36207 every 8 connection(s)
I20260810 15:16:11.211725 7798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:11.212668 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 7798
I20260810 15:16:11.212720 7912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:11.212750 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:11.212911 30616 external_mini_cluster.cc:1428] Setting key 107d048c9c67e28a3bedbc4cab12480c
I20260810 15:16:11.215777 7912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab: Bootstrap starting.
I20260810 15:16:11.216496 7912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:11.216774 7912 log.cc:824] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:11.217474 7912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab: No bootstrap required, opened a new log
I20260810 15:16:11.218407 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:11.213385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.29.230.62:36207)
0810 15:16:11.213583 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:11.213590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:11.213616 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:11.214087 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:11.214094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:11.214162 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:11.214274 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:11.214509 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.215466 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.216504 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.216975 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.217066 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:11.217710 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:11.217715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:11.217715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:11.217911 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:11.217954 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:11.217971 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:11.218040 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:11.218095 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:16:11.218177 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260810 15:16:11.219450 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:16:11.219827 7912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } }
I20260810 15:16:11.220013 7912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:11.220052 7912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35075730391b4ea48e19695413cf28ab, State: Initialized, Role: FOLLOWER
I20260810 15:16:11.220165 7912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } }
I20260810 15:16:11.220234 7912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:11.220257 7912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:11.220305 7912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:11.220860 7912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } }
I20260810 15:16:11.220958 7912 leader_election.cc:302] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35075730391b4ea48e19695413cf28ab; no voters:
I20260810 15:16:11.221086 7912 leader_election.cc:288] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:11.221148 7921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:11.221306 7921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [term 1 LEADER]: Becoming Leader. State: Replica: 35075730391b4ea48e19695413cf28ab, State: Running, Role: LEADER
I20260810 15:16:11.221318 7912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:11.221407 7921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } }
I20260810 15:16:11.221841 7923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35075730391b4ea48e19695413cf28ab. Latest consensus state: current_term: 1 leader_uuid: "35075730391b4ea48e19695413cf28ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } } }
I20260810 15:16:11.221873 7922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35075730391b4ea48e19695413cf28ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35075730391b4ea48e19695413cf28ab" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36207 } } }
I20260810 15:16:11.221938 7923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:11.221985 7922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:11.222157 7942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:11.222678 7942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:11.224320 7942 catalog_manager.cc:1347] Generated new cluster ID: b927e1f3a67e4b0ba37b7799e3b8e789
I20260810 15:16:11.224372 7942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:11.232409 7942 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:11.232928 7942 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:11.238059 7942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35075730391b4ea48e19695413cf28ab: Generated new TSK 0
I20260810 15:16:11.238225 7942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:11.674543 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:11.244452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.29.230.62:36207)
0810 15:16:11.244503 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:11.244505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:11.244517 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:16:11.244557 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:11.244557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:11.244569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:11.244639 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0810 15:16:11.244804 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.245440 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.246605 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.246810 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.246870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:11.247054 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 15:16:11.674301 (+427247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 15:16:11.674471 (+ 170us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260810 15:16:11.674620 7917 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51870: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 15:16:11.674706 7969 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:11.244250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36207 (local address 127.0.0.1:51870)
0810 15:16:11.244421 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:11.244433 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:11.244496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:11.244649 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:11.244650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:11.244652 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 15:16:11.244717 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:11.244720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.245538 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.245539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:11.246505 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:11.246509 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:11.246884 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:11.246884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:11.246945 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:11.246952 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 15:16:11.674468 (+427516us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 15:16:11.674609 (+ 141us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.230.62:36207: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260810 15:16:11.675038 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 7798
2026-08-10T15:16:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (611 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:11 dist-test-slave-gjrz krb5kdc[7990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:11 dist-test-slave-gjrz krb5kdc[7990](info): set up 2 sockets
Aug 10 15:16:11 dist-test-slave-gjrz krb5kdc[7990](info): commencing operation
krb5kdc: starting...
W20260810 15:16:13.053210 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:13 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
W20260810 15:16:14.396901 30616 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.230.62: real 1.241s user 0.000s sys 0.003s
I20260810 15:16:14.397296 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33969
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:14.481290 8016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:14.481585 8016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:14.481618 8016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:14.483414 8016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:14.483474 8016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:14.483493 8016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:14.483510 8016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:14.483533 8016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:14.485186 8016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:14.485412 8016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:14.485607 8016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:14.490434 8027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:14.490533 8016 server_base.cc:1034] running on GCE node
W20260810 15:16:14.490434 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:14.490447 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:14.490959 8016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:14.491344 8016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:14.492563 8016 hybrid_clock.cc:648] HybridClock initialized: now 1786374974492542 us; error 32 us; skew 500 ppm
Aug 10 15:16:14 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:14.495460 8016 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:14.496023 8016 webserver.cc:460] Webserver started at http://127.29.230.62:36887/ using document root <none> and password file <none>
I20260810 15:16:14.496302 8016 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:14.496366 8016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:14.496508 8016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:14.497972 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "a7833357ce1444098afdecb3dd98abbb"
format_stamp: "Formatted at 2026-08-10 15:16:14 on dist-test-slave-gjrz"
server_key: "8f5269ad3e5d9a4a7277f9ea85636336"
server_key_iv: "00d323b9f3ee66161f83512583704eaa"
server_key_version: "encryptionkey@0"
I20260810 15:16:14.498421 8016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "a7833357ce1444098afdecb3dd98abbb"
format_stamp: "Formatted at 2026-08-10 15:16:14 on dist-test-slave-gjrz"
server_key: "8f5269ad3e5d9a4a7277f9ea85636336"
server_key_iv: "00d323b9f3ee66161f83512583704eaa"
server_key_version: "encryptionkey@0"
I20260810 15:16:14.500254 8016 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 15:16:14.501186 8041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:14.501364 8016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:14.501475 8016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "a7833357ce1444098afdecb3dd98abbb"
format_stamp: "Formatted at 2026-08-10 15:16:14 on dist-test-slave-gjrz"
server_key: "8f5269ad3e5d9a4a7277f9ea85636336"
server_key_iv: "00d323b9f3ee66161f83512583704eaa"
server_key_version: "encryptionkey@0"
I20260810 15:16:14.501566 8016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:14.530254 8016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:14.533125 8016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:14.533289 8016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:14.537348 8016 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:33969
I20260810 15:16:14.537386 8134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:33969 every 8 connection(s)
I20260810 15:16:14.537735 8016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:14.538625 8138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:14.541102 8138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: Bootstrap starting.
I20260810 15:16:14.541703 8138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:14.541993 8138 log.cc:824] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:14.542719 8138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: No bootstrap required, opened a new log
I20260810 15:16:14.543367 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 8016
I20260810 15:16:14.543457 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:14.543620 30616 external_mini_cluster.cc:1428] Setting key a57843871477b060585dd3c0af49491c
I20260810 15:16:14.545315 8138 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } }
I20260810 15:16:14.545482 8138 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:14.545553 8138 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7833357ce1444098afdecb3dd98abbb, State: Initialized, Role: FOLLOWER
I20260810 15:16:14.545694 8138 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } }
I20260810 15:16:14.545781 8138 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:14.545820 8138 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:14.545871 8138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 0 FOLLOWER]: Advancing to term 1
Aug 10 15:16:14 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:14.553241 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:14.544172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.29.230.62:33969)
0810 15:16:14.544332 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:14.544339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:14.544365 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:14.545250 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:14.545257 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:14.545309 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:14.545426 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:14.546253 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:14.547161 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:14.547908 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:14.548306 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:14.548385 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:14.550464 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:14.550467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:14.550468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:14.550478 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:14.552034 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:14.552317 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:14.552320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:14.552320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:14.552398 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:14.552534 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:14.552534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:14.552535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:14.552771 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:14.552885 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:14.552939 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 15:16:14.553039 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260810 15:16:14.583812 8138 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } }
I20260810 15:16:14.584046 8138 leader_election.cc:302] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7833357ce1444098afdecb3dd98abbb; no voters:
I20260810 15:16:14.584290 8138 leader_election.cc:288] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:14.584465 8149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:14.584585 8138 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:14.584641 8149 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [term 1 LEADER]: Becoming Leader. State: Replica: a7833357ce1444098afdecb3dd98abbb, State: Running, Role: LEADER
I20260810 15:16:14.584774 8149 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } }
I20260810 15:16:14.585273 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7833357ce1444098afdecb3dd98abbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } } }
I20260810 15:16:14.585376 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:14.585544 8151 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7833357ce1444098afdecb3dd98abbb. Latest consensus state: current_term: 1 leader_uuid: "a7833357ce1444098afdecb3dd98abbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7833357ce1444098afdecb3dd98abbb" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33969 } } }
I20260810 15:16:14.585618 8151 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb [sys.catalog]: This master's current role is: LEADER
W20260810 15:16:14.586365 8180 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:14.586424 8180 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:14.586696 8166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:14.586874 8166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:14.588587 8166 catalog_manager.cc:1347] Generated new cluster ID: b8c03f35e4f64010a48412ab85dbcd5f
I20260810 15:16:14.588649 8166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:14.618108 8166 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:14.618670 8166 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:14.623102 8166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7833357ce1444098afdecb3dd98abbb: Generated new TSK 0
I20260810 15:16:14.623307 8166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
W20260810 15:16:14.982021 30616 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.230.1: real 0.356s user 0.000s sys 0.004s
I20260810 15:16:14.982383 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:15.063856 8189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:15.064066 8189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:15.064098 8189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:15.065511 8189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:15.065555 8189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:15.065580 8189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:15.067049 8189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:15.067287 8189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:15.067514 8189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:15.072352 8203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:15.072304 8199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:15.072553 8189 server_base.cc:1034] running on GCE node
W20260810 15:16:15.072300 8201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:15.072813 8189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:15.073033 8189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:15.074172 8189 hybrid_clock.cc:648] HybridClock initialized: now 1786374975074162 us; error 32 us; skew 500 ppm
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.076753 8189 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:15.077196 8189 webserver.cc:460] Webserver started at http://127.29.230.1:41365/ using document root <none> and password file <none>
I20260810 15:16:15.077391 8189 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:15.077440 8189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:15.077569 8189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:15.078572 8189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "16dcf4ee159b49728eb6ae6abeb9298d"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "c1a7edbae933f3459c9348494ab1fc8b"
server_key_iv: "997ef8920e5b76bf801930b10f7262d9"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.078971 8189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "16dcf4ee159b49728eb6ae6abeb9298d"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "c1a7edbae933f3459c9348494ab1fc8b"
server_key_iv: "997ef8920e5b76bf801930b10f7262d9"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.080814 8189 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 15:16:15.081770 8212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.081967 8189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:15.082077 8189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "16dcf4ee159b49728eb6ae6abeb9298d"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "c1a7edbae933f3459c9348494ab1fc8b"
server_key_iv: "997ef8920e5b76bf801930b10f7262d9"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.082161 8189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:15.123212 8189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:15.126132 8189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:15.126281 8189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:15.126523 8189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:15.126891 8189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:15.126935 8189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.126966 8189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:15.126991 8189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.133049 8189 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:43771
I20260810 15:16:15.133103 8433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:43771 every 8 connection(s)
I20260810 15:16:15.133436 8189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:15.141595 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 8189
I20260810 15:16:15.141690 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:15.141877 30616 external_mini_cluster.cc:1428] Setting key eb8dc790c319d96fb6b96263609bd6a1
I20260810 15:16:15.144901 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.134190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:42181 (local address 127.29.230.62:33969)
0810 15:16:15.134306 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.134308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.134508 (+ 200us) server_negotiation.cc:407] Connection header received
0810 15:16:15.135825 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.135826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.135841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.135907 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.136882 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.137645 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.139157 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.139452 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.139517 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.141994 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.141995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.141996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.142003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.143419 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.143860 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.143861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.143861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.143912 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.144196 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.144197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.144197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.144347 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.144418 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.144701 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.144802 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
I20260810 15:16:15.144923 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.134199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33969 (local address 127.29.230.1:42181)
0810 15:16:15.134364 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:15.134418 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:15.135688 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:15.136059 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:15.136065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:15.136451 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:15.136760 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.136767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.137744 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.137745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.139066 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.139069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.139591 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.139593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.139732 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.140215 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:15.140224 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:15.141868 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:15.143490 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.143499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.143501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.143791 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.143961 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.143962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.143963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.144114 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.144411 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:15.144415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:15.144539 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 15:16:15.144629 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:16:15.144758 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260810 15:16:15.145429 8435 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33969
I20260810 15:16:15.145550 8435 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:15.145728 8435 heartbeater.cc:511] Master 127.29.230.62:33969 requested a full tablet report, sending...
I20260810 15:16:15.146270 8070 ts_manager.cc:194] Registered new tserver with Master: 16dcf4ee159b49728eb6ae6abeb9298d (127.29.230.1:43771)
I20260810 15:16:15.147145 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:42181
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:15.167923 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:15.249466 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:15.249678 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:15.249709 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:15.250857 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:15.250898 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:15.250934 8443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:15.252581 8443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:15.252854 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:15.253132 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:15.258222 8457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:15.258325 8443 server_base.cc:1034] running on GCE node
W20260810 15:16:15.258229 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:15.258433 8456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:15.258656 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:15.258868 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:15.259997 8443 hybrid_clock.cc:648] HybridClock initialized: now 1786374975259984 us; error 29 us; skew 500 ppm
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.262450 8443 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:15.262879 8443 webserver.cc:460] Webserver started at http://127.29.230.2:46181/ using document root <none> and password file <none>
I20260810 15:16:15.263092 8443 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:15.263146 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:15.263306 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:15.264374 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "d49396d63d2b44f18896da1f2a308960"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "30e03b225da817ea648deee01e7d3fe2"
server_key_iv: "bb9b7541ff937f401a1f0a3173778a09"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.264705 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "d49396d63d2b44f18896da1f2a308960"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "30e03b225da817ea648deee01e7d3fe2"
server_key_iv: "bb9b7541ff937f401a1f0a3173778a09"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.265923 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:16:15.266608 8476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.266763 8443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:15.266844 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "d49396d63d2b44f18896da1f2a308960"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "30e03b225da817ea648deee01e7d3fe2"
server_key_iv: "bb9b7541ff937f401a1f0a3173778a09"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.266898 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:15.277314 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:15.279215 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:15.279371 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:15.279580 8443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:15.279860 8443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:15.279891 8443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.279912 8443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:15.279928 8443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.285691 8443 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:44635
I20260810 15:16:15.285737 8693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:44635 every 8 connection(s)
I20260810 15:16:15.286084 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:15.293336 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 8443
I20260810 15:16:15.293466 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:15.293654 30616 external_mini_cluster.cc:1428] Setting key 1aca110877823dc04ea7c4ca345715c8
I20260810 15:16:15.296020 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.286884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:54665 (local address 127.29.230.62:33969)
0810 15:16:15.286935 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.286937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.287202 (+ 265us) server_negotiation.cc:407] Connection header received
0810 15:16:15.288205 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.288207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.288217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.288257 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.288853 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.289354 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.290472 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.290762 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.290838 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.293020 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.293022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.293022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.293028 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.294653 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.295036 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.295038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.295038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.295101 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.295368 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.295369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.295369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.295490 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.295549 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.295824 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.295928 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:16:15.296185 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.286826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33969 (local address 127.29.230.2:54665)
0810 15:16:15.287089 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:15.287121 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:15.288099 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:15.288318 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:15.288321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:15.288573 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:15.288782 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.288786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.289459 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.289460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.290342 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.290345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.290884 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.290885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.291011 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.291563 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:15.291572 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:15.292931 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:15.294732 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.294738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.294741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.294966 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.295157 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.295158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.295159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.295313 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.295550 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:15.295557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:15.295695 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 15:16:15.295790 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 15:16:15.295959 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
I20260810 15:16:15.296854 8698 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33969
I20260810 15:16:15.296949 8698 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:15.297140 8698 heartbeater.cc:511] Master 127.29.230.62:33969 requested a full tablet report, sending...
I20260810 15:16:15.297529 8070 ts_manager.cc:194] Registered new tserver with Master: d49396d63d2b44f18896da1f2a308960 (127.29.230.2:44635)
I20260810 15:16:15.297892 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:54665
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:15.320135 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:15.402216 8711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:15.402432 8711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:15.402464 8711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:15.403684 8711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:15.403729 8711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:15.403764 8711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:15.405328 8711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:15.405553 8711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:15.405748 8711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:15.410619 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:15.410763 8720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:15.410648 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:15.410881 8711 server_base.cc:1034] running on GCE node
I20260810 15:16:15.411175 8711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:15.411429 8711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:15.412591 8711 hybrid_clock.cc:648] HybridClock initialized: now 1786374975412584 us; error 37 us; skew 500 ppm
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.415146 8711 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:15.415630 8711 webserver.cc:460] Webserver started at http://127.29.230.3:46845/ using document root <none> and password file <none>
I20260810 15:16:15.415824 8711 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:15.415865 8711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:15.415957 8711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:15.416971 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "0d5bbf96f0304d9c938bb6911d4632e2"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "cafc7a15fb93cfb5e3d3ad9784c74231"
server_key_iv: "85fe90bb2f471df9deb5bb5316609cdf"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.417282 8711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "0d5bbf96f0304d9c938bb6911d4632e2"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "cafc7a15fb93cfb5e3d3ad9784c74231"
server_key_iv: "85fe90bb2f471df9deb5bb5316609cdf"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.418478 8711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:15.419166 8733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.419356 8711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:15.419454 8711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "0d5bbf96f0304d9c938bb6911d4632e2"
format_stamp: "Formatted at 2026-08-10 15:16:15 on dist-test-slave-gjrz"
server_key: "cafc7a15fb93cfb5e3d3ad9784c74231"
server_key_iv: "85fe90bb2f471df9deb5bb5316609cdf"
server_key_version: "encryptionkey@0"
I20260810 15:16:15.419523 8711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:15.434486 8711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:15.436494 8711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:15.436609 8711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:15.436798 8711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:15.437075 8711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:15.437109 8711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.437131 8711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:15.437147 8711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:15.442631 8711 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36141
I20260810 15:16:15.442700 8953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36141 every 8 connection(s)
I20260810 15:16:15.442977 8711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:15.445765 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 8711
I20260810 15:16:15.445855 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:15.446074 30616 external_mini_cluster.cc:1428] Setting key e0d6503fd1b9e59fc9f987bdaeed681b
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:15.452302 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.443760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:39467 (local address 127.29.230.62:33969)
0810 15:16:15.443816 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.443818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.444028 (+ 210us) server_negotiation.cc:407] Connection header received
0810 15:16:15.445008 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.445010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.445023 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.445075 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.445744 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.446336 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.447620 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.447821 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.447877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.449953 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.449955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.449955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.449962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.451080 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.451503 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.451503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.451504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.451556 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.451813 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.451814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.451814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.451911 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.451956 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.452177 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.452238 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:16:15.452500 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.443724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33969 (local address 127.29.230.3:39467)
0810 15:16:15.443897 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:15.443941 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:15.444894 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:15.445149 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:15.445153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:15.445445 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:15.445647 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.445652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.446449 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.446450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.447532 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:15.447535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.447898 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.447899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:15.448048 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.448481 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:15.448490 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:15.449872 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:15.451153 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.451159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.451161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.451446 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.451632 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:15.451634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:15.451634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:15.451755 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:15.451959 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:15.451964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:15.452083 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 15:16:15.452167 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 15:16:15.452293 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260810 15:16:15.453040 8960 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33969
I20260810 15:16:15.453128 8960 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:15.453277 8960 heartbeater.cc:511] Master 127.29.230.62:33969 requested a full tablet report, sending...
I20260810 15:16:15.453655 8070 ts_manager.cc:194] Registered new tserver with Master: 0d5bbf96f0304d9c938bb6911d4632e2 (127.29.230.3:36141)
I20260810 15:16:15.454041 8070 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:39467
I20260810 15:16:15.457808 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/test-admin.keytab.
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.473697 30616 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 15:16:15.475339 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:15.474578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.29.230.1:43771)
0810 15:16:15.474772 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.474777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.474788 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:15.474816 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.474818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.474832 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.474897 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.475237 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54386: BlockingRecv error: recv got EOF from 127.0.0.1:54386 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":107,"threads_started":1}
W20260810 15:16:15.475502 30616 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.476506 30616 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.477346 30616 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:16:15.484803 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.477922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.29.230.1:43771)
0810 15:16:15.477973 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.477975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.477984 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:15.478048 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.478048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.478058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.478107 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.478451 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.479204 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.479937 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.480277 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.480332 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.482197 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.482205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.482206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.482212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.483727 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.484012 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.484016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.484016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.484076 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.484232 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.484232 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.484233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.484380 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.484532 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.484636 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.484704 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 15:16:15.486552 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:15.486111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.29.230.1:43771)
0810 15:16:15.486148 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.486149 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.486171 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:16:15.486237 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.486238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.486246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.486280 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.486512 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54404: BlockingRecv error: recv got EOF from 127.0.0.1:54404 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:16:15.486672 30616 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.487648 30616 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.488590 30616 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:16:15.495780 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.489103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.29.230.1:43771)
0810 15:16:15.489151 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.489153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.489175 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:16:15.489193 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.489194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.489203 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.489244 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.489592 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.490068 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.490717 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.490937 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.490990 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.493148 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.493149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.493149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.493156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.494410 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.494864 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.494866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.494866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.494926 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.495192 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.495194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.495194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.495353 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.495462 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.495608 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.495691 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260810 15:16:15.497375 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:15.496819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.29.230.1:43771)
0810 15:16:15.496857 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.496858 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.496920 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:16:15.496979 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.496980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.496988 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.497033 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.497331 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54426: BlockingRecv error: recv got EOF from 127.0.0.1:54426 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:16:15.497380 30616 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786374947162751-30616-0/krb5kdc/krb5cc' of type FILE
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.498504 30616 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:15.499572 30616 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[7990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:16:15.506165 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:15.500165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.29.230.1:43771)
0810 15:16:15.500220 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:15.500221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:15.500230 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:15.500285 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:15.500286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:15.500295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:15.500338 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:15.500674 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.501140 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.501815 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:15.502029 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:15.502080 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:15.503996 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:15.503997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:15.503997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:15.504004 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:15.505180 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.505512 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.505513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.505513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.505568 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:15.505722 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:15.505723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:15.505723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:15.505819 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:15.505916 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:15.506001 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:16:15.506075 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:16:15.506821 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 8189
I20260810 15:16:15.519795 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 8443
I20260810 15:16:15.538621 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 8711
I20260810 15:16:15.551853 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 8016
2026-08-10T15:16:15Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3692 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[9000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[9000](info): set up 2 sockets
Aug 10 15:16:15 dist-test-slave-gjrz krb5kdc[9000](info): commencing operation
krb5kdc: starting...
W20260810 15:16:16.735988 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:16 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab.
Entry for principal oryx/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab.
I20260810 15:16:16.854337 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37971
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab
--principal=oryx/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:37971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:16.936405 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:16.936599 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:16.936619 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:16.937690 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:16.937721 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:16.937732 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:16.937743 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:16.937753 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:16.939167 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:37971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab
--principal=oryx/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:16.939411 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:16.939594 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:16.944427 9044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:16.944443 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:16.944478 9030 server_base.cc:1034] running on GCE node
W20260810 15:16:16.944469 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:16.944931 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:16.945257 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:16.946415 9030 hybrid_clock.cc:648] HybridClock initialized: now 1786374976946385 us; error 44 us; skew 500 ppm
Aug 10 15:16:16 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:16.951874 9030 init.cc:377] Logged in from keytab as oryx/127.29.230.62@KRBTEST.COM (short username oryx)
I20260810 15:16:16.952299 9030 webserver.cc:460] Webserver started at http://127.29.230.62:37403/ using document root <none> and password file <none>
I20260810 15:16:16.952518 9030 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:16.952564 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:16.952692 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:16.953725 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42"
format_stamp: "Formatted at 2026-08-10 15:16:16 on dist-test-slave-gjrz"
server_key: "28ee4d61f19f162ccda1b58aa903fb8e"
server_key_iv: "c9c21c5a83e469f4e3bc7d55e532d9a5"
server_key_version: "encryptionkey@0"
I20260810 15:16:16.954064 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42"
format_stamp: "Formatted at 2026-08-10 15:16:16 on dist-test-slave-gjrz"
server_key: "28ee4d61f19f162ccda1b58aa903fb8e"
server_key_iv: "c9c21c5a83e469f4e3bc7d55e532d9a5"
server_key_version: "encryptionkey@0"
I20260810 15:16:16.955336 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:16.956054 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:16.956236 9030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:16.956328 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42"
format_stamp: "Formatted at 2026-08-10 15:16:16 on dist-test-slave-gjrz"
server_key: "28ee4d61f19f162ccda1b58aa903fb8e"
server_key_iv: "c9c21c5a83e469f4e3bc7d55e532d9a5"
server_key_version: "encryptionkey@0"
I20260810 15:16:16.956394 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:16.965045 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:16.966781 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:16.966884 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:16.970706 9030 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:37971
I20260810 15:16:16.970748 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:37971 every 8 connection(s)
I20260810 15:16:16.971050 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:16.972015 9160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:16.974305 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Bootstrap starting.
I20260810 15:16:16.974869 9160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:16.975133 9160 log.cc:824] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:16.975849 9160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: No bootstrap required, opened a new log
I20260810 15:16:16.977556 9160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } }
I20260810 15:16:16.977706 9160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:16.977741 9160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3b0d6e183f4d08b5ad1f64f18b9e42, State: Initialized, Role: FOLLOWER
I20260810 15:16:16.977891 9160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } }
I20260810 15:16:16.977991 9160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:16.978026 9160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:16.978075 9160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:16.978894 9160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } }
I20260810 15:16:16.979007 9160 leader_election.cc:302] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a3b0d6e183f4d08b5ad1f64f18b9e42; no voters:
I20260810 15:16:16.979178 9160 leader_election.cc:288] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:16.979212 9166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:16.979390 9166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 LEADER]: Becoming Leader. State: Replica: 2a3b0d6e183f4d08b5ad1f64f18b9e42, State: Running, Role: LEADER
I20260810 15:16:16.979473 9160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:16.979491 9166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } }
I20260810 15:16:16.979830 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3b0d6e183f4d08b5ad1f64f18b9e42. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } }
I20260810 15:16:16.979914 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:16.979974 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 9030
I20260810 15:16:16.980121 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:16.980278 9179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:16.980384 30616 external_mini_cluster.cc:1428] Setting key 02c4674bdbb53c06e78b9fa08329d1a4
I20260810 15:16:16.980911 9167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } }
I20260810 15:16:16.981060 9167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:16.981068 9179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:16.982800 9179 catalog_manager.cc:1347] Generated new cluster ID: eba84c09f8ad41b6976183e903c3381c
I20260810 15:16:16.982861 9179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:16:16 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:16.989931 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:16.980934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.29.230.62:37971)
0810 15:16:16.981341 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:16.981351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:16.981383 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:16:16.981506 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:16.981513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:16.981557 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:16.981679 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:16.982598 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:16.983690 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:16.984415 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:16.984851 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:16.984946 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:16.987051 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:16.987056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:16.987056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:16.987068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:16.988553 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:16.988870 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:16.988874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:16.988874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:16.988970 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:16.989150 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:16.989151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:16.989151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:16.989374 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:16.989531 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:16.989608 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 15:16:16.989746 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":65,"threads_started":1}
I20260810 15:16:16.997066 9179 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:16.997727 9179 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:17.007869 9179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Generated new TSK 0
I20260810 15:16:17.008159 9179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
I20260810 15:16:17.041392 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:17.123466 9217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:17.123673 9217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:17.123704 9217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:17.124852 9217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:17.124893 9217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:17.124928 9217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:17.126358 9217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:17.126595 9217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:17.126798 9217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.131551 9232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.131557 9228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.131557 9230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.131897 9217 server_base.cc:1034] running on GCE node
I20260810 15:16:17.132081 9217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.132293 9217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.133428 9217 hybrid_clock.cc:648] HybridClock initialized: now 1786374977133417 us; error 32 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.135880 9217 init.cc:377] Logged in from keytab as oryx/127.29.230.1@KRBTEST.COM (short username oryx)
I20260810 15:16:17.136368 9217 webserver.cc:460] Webserver started at http://127.29.230.1:34773/ using document root <none> and password file <none>
I20260810 15:16:17.136574 9217 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.136622 9217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.136744 9217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:17.137771 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "8359f4ef0b1d4e9083872a01426bc7cf"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "aa6fb1b4711f4763e369d546202aafec"
server_key_iv: "2fb04bdeb58c68be0491394c8c43d2a3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.138106 9217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "8359f4ef0b1d4e9083872a01426bc7cf"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "aa6fb1b4711f4763e369d546202aafec"
server_key_iv: "2fb04bdeb58c68be0491394c8c43d2a3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.139381 9217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:16:17.140070 9245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.140226 9217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.140309 9217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "8359f4ef0b1d4e9083872a01426bc7cf"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "aa6fb1b4711f4763e369d546202aafec"
server_key_iv: "2fb04bdeb58c68be0491394c8c43d2a3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.140375 9217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.149211 9217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.150975 9217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:17.151113 9217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.151330 9217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:17.151613 9217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:17.151645 9217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.151672 9217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:17.151697 9217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.157172 9217 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44389
I20260810 15:16:17.157230 9474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44389 every 8 connection(s)
I20260810 15:16:17.157501 9217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:17.166190 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.158169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:51797 (local address 127.29.230.62:37971)
0810 15:16:17.158223 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.158225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.158421 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:16:17.158899 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.158900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.158914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.158963 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.159929 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.160440 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.161344 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.161541 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.161586 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.163558 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.163559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.163559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.163566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.164968 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.165354 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.165355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.165355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.165407 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.165673 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.165674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.165674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.165776 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.165821 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.166035 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.166112 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:16:17.166365 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.158139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:51797)
0810 15:16:17.158307 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.158342 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.158819 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.159026 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.159029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.159672 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.159861 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.159866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.160508 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.160509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.161285 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.161287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.161611 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.161612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.161716 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.162165 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.162173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.163472 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.165060 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.165066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.165068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.165287 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.165492 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.165493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.165494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.165611 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.165822 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.165825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.165936 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.166047 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.166173 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260810 15:16:17.166782 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 9217
I20260810 15:16:17.166884 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:17.166875 9475 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:17.166965 9475 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:17.167083 30616 external_mini_cluster.cc:1428] Setting key 80459b9e5b356d49c943ff6c0a0085c6
I20260810 15:16:17.167196 9475 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:17.167809 9085 ts_manager.cc:194] Registered new tserver with Master: 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1:44389)
I20260810 15:16:17.168411 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:51797
WARNING: no policy specified for oryx/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
I20260810 15:16:17.192169 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:17.275348 9484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:17.275544 9484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:17.275568 9484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:17.276751 9484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:17.276793 9484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:17.276818 9484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:17.278215 9484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:17.278391 9484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:17.278571 9484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.283618 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.283594 9493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.283667 9484 server_base.cc:1034] running on GCE node
W20260810 15:16:17.283588 9498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.284013 9484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.284240 9484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.285390 9484 hybrid_clock.cc:648] HybridClock initialized: now 1786374977285381 us; error 31 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.287866 9484 init.cc:377] Logged in from keytab as oryx/127.29.230.2@KRBTEST.COM (short username oryx)
I20260810 15:16:17.288321 9484 webserver.cc:460] Webserver started at http://127.29.230.2:36923/ using document root <none> and password file <none>
I20260810 15:16:17.288533 9484 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.288576 9484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.288693 9484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:17.289737 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "2317b4eb6d794b208ca390c7924312cb"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "925f068a4142c8c78a18875ca11b4569"
server_key_iv: "b8e59167012c1ef9418e0473174f4203"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.290076 9484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "2317b4eb6d794b208ca390c7924312cb"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "925f068a4142c8c78a18875ca11b4569"
server_key_iv: "b8e59167012c1ef9418e0473174f4203"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.291358 9484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:16:17.292064 9510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.292205 9484 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:17.292289 9484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "2317b4eb6d794b208ca390c7924312cb"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "925f068a4142c8c78a18875ca11b4569"
server_key_iv: "b8e59167012c1ef9418e0473174f4203"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.292356 9484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.303596 9484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.305526 9484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:17.305656 9484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.305873 9484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:17.306155 9484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:17.306188 9484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.306216 9484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:17.306233 9484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.311826 9484 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:46197
I20260810 15:16:17.311887 9721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:46197 every 8 connection(s)
I20260810 15:16:17.312163 9484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:17.317230 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 9484
I20260810 15:16:17.317315 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:17.317477 30616 external_mini_cluster.cc:1428] Setting key b8752ca06b68e2eda032ad768b316f43
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:17.321089 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.312840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:42701 (local address 127.29.230.62:37971)
0810 15:16:17.312928 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.312930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.313110 (+ 180us) server_negotiation.cc:407] Connection header received
0810 15:16:17.313590 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.313591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.313602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.313652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.314564 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.315078 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.316118 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.316316 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.316363 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.318556 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.318557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.318558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.318571 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.319812 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.320199 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.320201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.320201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.320256 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.320480 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.320481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.320481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.320588 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.320644 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.320901 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.321000 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 15:16:17.321237 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.312759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:42701)
0810 15:16:17.313002 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.313038 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.313508 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.313722 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.313725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.314280 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.314493 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.314498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.315175 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.315177 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.316051 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.316054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.316376 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.316377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.316493 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.316968 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.316977 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.318462 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.319887 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.319894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.319896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.320119 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.320311 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.320312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.320312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.320416 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.320644 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.320650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.320784 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.320873 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.321030 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260810 15:16:17.321702 9723 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:17.321801 9723 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:17.322001 9723 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:17.322456 9085 ts_manager.cc:194] Registered new tserver with Master: 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
I20260810 15:16:17.322835 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:42701
WARNING: no policy specified for oryx/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
I20260810 15:16:17.342337 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:17.423154 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:17.423360 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:17.423382 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:17.424458 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:17.424489 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:17.424513 9730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:17.425894 9730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:37971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:17.426077 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:17.426254 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.431108 9736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.431113 9740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.431121 9735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.431197 9730 server_base.cc:1034] running on GCE node
I20260810 15:16:17.431499 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.431731 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.432894 9730 hybrid_clock.cc:648] HybridClock initialized: now 1786374977432865 us; error 39 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.435331 9730 init.cc:377] Logged in from keytab as oryx/127.29.230.3@KRBTEST.COM (short username oryx)
I20260810 15:16:17.435783 9730 webserver.cc:460] Webserver started at http://127.29.230.3:36303/ using document root <none> and password file <none>
I20260810 15:16:17.436033 9730 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.436080 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.436187 9730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:17.437203 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "f8022693fa844a238473161008ed0b49"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "62305ffdccc5d88767f2406eee72b1d0"
server_key_iv: "ab79e1787d3ce8aa76b2e9f09824d95b"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.437660 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "f8022693fa844a238473161008ed0b49"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "62305ffdccc5d88767f2406eee72b1d0"
server_key_iv: "ab79e1787d3ce8aa76b2e9f09824d95b"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.438937 9730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:17.439687 9749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.439850 9730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.439921 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "f8022693fa844a238473161008ed0b49"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "62305ffdccc5d88767f2406eee72b1d0"
server_key_iv: "ab79e1787d3ce8aa76b2e9f09824d95b"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.439975 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.453771 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.455724 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:17.455837 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.456038 9730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:17.456338 9730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:17.456372 9730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.456395 9730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:17.456406 9730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.461917 9730 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:33353
I20260810 15:16:17.461972 9967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:33353 every 8 connection(s)
I20260810 15:16:17.462271 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:17.467468 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 9730
I20260810 15:16:17.467567 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:17.467772 30616 external_mini_cluster.cc:1428] Setting key 481a75d7e6eff2ad4dd86a44c4589bfa
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:17.471031 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.462957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:47717 (local address 127.29.230.62:37971)
0810 15:16:17.462998 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.463000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.463212 (+ 212us) server_negotiation.cc:407] Connection header received
0810 15:16:17.463741 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.463742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.463751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.463804 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.464683 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.465188 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.466165 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.466346 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.466397 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.468630 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.468632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.468632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.468640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.469757 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.470149 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.470150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.470151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.470209 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.470418 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.470419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.470420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.470544 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.470602 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.470856 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.470933 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:16:17.471177 9973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.462930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:47717)
0810 15:16:17.463098 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.463141 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.463657 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.463859 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.463862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.464420 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.464614 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.464618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.465263 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.465264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.466096 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.466098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.466404 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.466405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.466501 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.466972 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.466980 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.468532 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.469855 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.469862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.469864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.470080 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.470263 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.470264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.470264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.470361 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.470604 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.470611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.470736 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.470823 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.470973 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260810 15:16:17.471628 9968 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:17.471716 9968 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:17.471882 9968 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:17.472261 9085 ts_manager.cc:194] Registered new tserver with Master: f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:17.472635 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:47717
I20260810 15:16:17.479508 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.230.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.61.keytab.
Entry for principal oryx/127.29.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.61.keytab.
I20260810 15:16:17.506115 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:35991
--webserver_interface=127.29.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.61.keytab
--principal=oryx/127.29.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:37971,127.29.230.61:35991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:17.587139 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:17.587364 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:17.587396 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:17.588527 9982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:17.588567 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:17.588584 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:17.588600 9982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:17.588615 9982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:17.590091 9982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:37971,127.29.230.61:35991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.61.keytab
--principal=oryx/127.29.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
W20260810 15:16:17.590209 9982 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.230.62:37971,127.29.230.61:35991'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 15:16:17.590353 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:17.590564 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.595176 9994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.595326 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.595176 9992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.595358 9982 server_base.cc:1034] running on GCE node
I20260810 15:16:17.595647 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.595933 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.597103 9982 hybrid_clock.cc:648] HybridClock initialized: now 1786374977597081 us; error 37 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.599578 9982 init.cc:377] Logged in from keytab as oryx/127.29.230.61@KRBTEST.COM (short username oryx)
I20260810 15:16:17.600041 9982 webserver.cc:460] Webserver started at http://127.29.230.61:33525/ using document root <none> and password file <none>
I20260810 15:16:17.600260 9982 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.600306 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.600431 9982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:17.601449 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/instance:
uuid: "1966bcfcdb48443eaf4676295f979e71"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "a12044874f80d72a65b96c04e31d1e46"
server_key_iv: "09831796dbc9b2c5883187c8bf130098"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.601840 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal/instance:
uuid: "1966bcfcdb48443eaf4676295f979e71"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "a12044874f80d72a65b96c04e31d1e46"
server_key_iv: "09831796dbc9b2c5883187c8bf130098"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.603116 9982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:17.603885 10002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.604027 9982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.604112 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "1966bcfcdb48443eaf4676295f979e71"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "a12044874f80d72a65b96c04e31d1e46"
server_key_iv: "09831796dbc9b2c5883187c8bf130098"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.604173 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.616497 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.618556 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:17.618690 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.622622 9982 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.61:35991
I20260810 15:16:17.622655 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.61:35991 every 8 connection(s)
I20260810 15:16:17.622989 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
I20260810 15:16:17.623981 10100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:17.625443 10100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.61@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:17.631372 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 9982
I20260810 15:16:17.634450 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.626085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.29.230.62:37971)
0810 15:16:17.626157 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.626159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.626351 (+ 192us) server_negotiation.cc:407] Connection header received
0810 15:16:17.626828 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.626830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.626841 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.626903 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.627532 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.628037 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.629049 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.629249 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.629315 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.631425 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.631427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.631428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.631438 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.633191 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.633539 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.633540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.633541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.633603 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.633787 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.633788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.633788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.633965 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.634022 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.634319 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.634386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:16:17.634666 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.626053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57786)
0810 15:16:17.626236 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.626273 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.626721 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.626976 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.626979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.627212 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.627455 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.627462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.628122 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.628123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.628936 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.628938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.629352 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.629353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.629476 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.629930 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.629939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.631338 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.633265 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.633271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.633272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.633474 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.633661 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.633662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.633662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.633745 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.634026 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.634032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.634182 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.634297 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.634450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260810 15:16:17.635347 10100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.61@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:17.640285 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.631964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.29.230.61:35991)
0810 15:16:17.632128 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.632132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.632213 (+ 81us) server_negotiation.cc:407] Connection header received
0810 15:16:17.632262 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.632264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.632278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.632355 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.632898 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.633823 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.634854 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.635191 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.635281 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.637561 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.637567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.637567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.637571 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.639094 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.639474 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.639477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.639477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.639560 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.639742 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.639742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.639743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.639865 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.639978 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.640150 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.640218 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260810 15:16:17.641845 10105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.635597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57838)
0810 15:16:17.635649 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.635657 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.635689 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.635890 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.635891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.636032 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.636088 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.636090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.636678 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.636679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.637142 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.637144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.637479 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.637481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.637556 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.637868 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.637872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.639125 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.641191 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.641191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.641192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.641338 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.641471 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.641471 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.641471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.641515 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.641661 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.641662 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.641691 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.641728 (+ 37us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.641791 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 15:16:17.641964 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.635652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57838 (local address 127.29.230.61:35991)
0810 15:16:17.635773 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.635774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.635781 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:17.635799 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.635800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.635809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.635865 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.636159 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.636611 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.637220 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.637412 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.637453 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.639205 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.639206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.639206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.639212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.641130 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.641392 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.641393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.641393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.641437 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.641546 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.641546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.641546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.641628 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.641670 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.641750 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.641832 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260810 15:16:17.643186 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Bootstrap starting.
I20260810 15:16:17.643771 10100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:17.644040 10100 log.cc:824] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:17.644649 10100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: No bootstrap required, opened a new log
I20260810 15:16:17.645925 10100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } }
I20260810 15:16:17.646065 10100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:17.646088 10100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1966bcfcdb48443eaf4676295f979e71, State: Initialized, Role: FOLLOWER
I20260810 15:16:17.646181 10100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } }
I20260810 15:16:17.646453 10100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:17.646414 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } } }
I20260810 15:16:17.646514 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:16:17.648044 10139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:17.648103 10139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:17.649740 9982 master_runner.cc:428] Detected that this master 1966bcfcdb48443eaf4676295f979e71 is joining an existing cluster
I20260810 15:16:17.649827 9982 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.230.61:35991
I20260810 15:16:17.657018 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.650797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57854)
0810 15:16:17.650969 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.650979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.651037 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.651175 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.651176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.651430 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.651519 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.651521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.652420 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.652421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.653308 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.653311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.653733 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.653791 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.654095 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.654099 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.654449 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.656018 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.656019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.656280 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.656455 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.656456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.656501 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.656729 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.656730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.656752 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.656803 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.656918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":44,"thread_start_us":110,"threads_started":1}
I20260810 15:16:17.657194 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.650851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.29.230.62:37971)
0810 15:16:17.650892 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.650894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.651005 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:16:17.651099 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.651100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.651110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.651161 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.651650 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.652405 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653089 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.653282 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653342 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.654404 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.654406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.654406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.654414 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.656001 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656371 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.656372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.656372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.656436 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656623 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.656623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.656624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.656750 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.656815 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.657025 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.657110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:17.657121 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.650957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.29.230.61:35991)
0810 15:16:17.650986 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.650987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.651024 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:16:17.651093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.651094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.651101 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.651125 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.651588 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.652335 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653409 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.653667 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653724 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.654511 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.654512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.654512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.654518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.655943 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656355 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.656356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.656357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.656403 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656558 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.656559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.656559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.656669 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.656702 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.656832 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.657005 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":16}
I20260810 15:16:17.657466 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.650797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57796)
0810 15:16:17.650942 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.650952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.651031 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.651164 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.651165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.651427 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.651532 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.651534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.652469 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.652470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.653016 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.653018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.653353 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.653354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.653422 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.653921 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.653924 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.654316 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.656077 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.656078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.656309 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.656500 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.656500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.656500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.656567 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.656861 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.656861 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.656903 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.657263 (+ 360us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.657363 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":22,"thread_start_us":93,"threads_started":1}
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.62@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:17.666800 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.659818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57860)
0810 15:16:17.659958 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.659999 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.660063 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.660194 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.660196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.660448 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.660494 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.660496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.661322 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.661323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.661908 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.661910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.662333 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.662334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.662402 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.662797 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.662802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.664276 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.665822 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.665829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.665831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.666031 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.666212 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.666212 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.666213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.666261 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.666474 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.666491 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.666545 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.666625 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.666732 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260810 15:16:17.666808 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.659865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.29.230.61:35991)
0810 15:16:17.659896 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.659898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.660047 (+ 149us) server_negotiation.cc:407] Connection header received
0810 15:16:17.660117 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.660117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.660127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.660190 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.660555 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.661245 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.661977 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.662267 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.662322 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.664356 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.664357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.664358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.664364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.665731 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.666098 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.666099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.666099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.666155 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.666307 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.666308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.666308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.666424 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.666483 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.666653 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.666723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:16:17.667326 9085 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.230.61:35991: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1966bcfcdb48443eaf4676295f979e71"
member_type: NON_VOTER
last_known_addr {
host: "127.29.230.61"
port: 35991
}
attrs {
promote: true
}
}
dest_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42"
cas_config_opid_index: -1
I20260810 15:16:17.667696 9085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } }
I20260810 15:16:17.668164 10158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } }
I20260810 15:16:17.668486 10047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:17.669179 10158 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.230.61:35991
I20260810 15:16:17.669221 10047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 15:16:17.669479 9982 master.cc:537] Master@127.29.230.61:35991 shutting down...
I20260810 15:16:17.669535 10158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 15:16:17.669921 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } }
I20260810 15:16:17.670284 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
W20260810 15:16:17.671234 9059 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 15:16:17.671646 9059 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 -> Peer 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61:35991): Couldn't send request to peer 1966bcfcdb48443eaf4676295f979e71. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 15:16:17.672032 9982 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:17.672132 9982 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:17.672178 9982 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: stopping tablet replica
I20260810 15:16:17.685351 9982 master.cc:552] Master@127.29.230.61:35991 shutdown complete.
I20260810 15:16:17.685425 9982 master_runner.cc:305] Clearing existing system tablet
I20260810 15:16:17.685544 9982 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.685587 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.686208 9982 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.002s
I20260810 15:16:17.686638 10165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.686784 9982 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.686839 9982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.686862 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "1966bcfcdb48443eaf4676295f979e71"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "a12044874f80d72a65b96c04e31d1e46"
server_key_iv: "09831796dbc9b2c5883187c8bf130098"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.687623 9982 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:17.688864 9982 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 15:16:17.688926 9982 log.cc:1196] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 15:16:17.689108 9982 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Deleting consensus metadata
I20260810 15:16:17.689227 9982 master_runner.cc:315] Copying system tablet from 127.29.230.62:37971
I20260810 15:16:17.689810 9982 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: tablet copy: Beginning tablet copy session from remote peer at address 127.29.230.62:37971
I20260810 15:16:17.696341 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.690241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.29.230.62:37971)
0810 15:16:17.690288 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.690290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.690415 (+ 125us) server_negotiation.cc:407] Connection header received
0810 15:16:17.690474 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.690474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.690485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.690528 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.690840 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.691504 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.692557 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.692748 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.692801 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.693848 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.693850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.693850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.693857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.695241 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.695657 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.695658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.695658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.695714 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.695885 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.695885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.695885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.696008 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.696055 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.696217 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.696279 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:16:17.696409 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.690182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57804)
0810 15:16:17.690343 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.690352 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.690417 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.690533 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.690533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.690709 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.690764 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.690766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.691614 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.691615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.692476 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.692479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.692815 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.692816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.692886 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.693295 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.693300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.693696 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.695339 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.695341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.695341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.695583 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.695768 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.695769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.695770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.695832 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.696077 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.696078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.696125 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.696200 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.696304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260810 15:16:17.696789 9129 tablet_copy_service.cc:140] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1966bcfcdb48443eaf4676295f979e71 ({username='oryx', principal='oryx/127.29.230.61@KRBTEST.COM'} at 127.0.0.1:57804)
I20260810 15:16:17.696897 9129 tablet_copy_service.cc:161] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1966bcfcdb48443eaf4676295f979e71 at {username='oryx', principal='oryx/127.29.230.61@KRBTEST.COM'} at 127.0.0.1:57804: session id = 1966bcfcdb48443eaf4676295f979e71-00000000000000000000000000000000
I20260810 15:16:17.697801 9129 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 15:16:17.698401 9982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 15:16:17.699409 9982 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: tablet copy: Starting download of 0 data blocks...
I20260810 15:16:17.699542 9982 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: tablet copy: Starting download of 1 WAL segments...
I20260810 15:16:17.700565 9982 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 15:16:17.701314 9129 tablet_copy_service.cc:342] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Request end of tablet copy session 1966bcfcdb48443eaf4676295f979e71-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.230.61@KRBTEST.COM'} at 127.0.0.1:57804
I20260810 15:16:17.701362 9129 tablet_copy_service.cc:433] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: ending tablet copy session 1966bcfcdb48443eaf4676295f979e71-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1966bcfcdb48443eaf4676295f979e71
I20260810 15:16:17.703653 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.704700 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.704756 10180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.704998 10183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.705006 9982 server_base.cc:1034] running on GCE node
I20260810 15:16:17.705125 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.705256 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.706377 9982 hybrid_clock.cc:648] HybridClock initialized: now 1786374977706365 us; error 26 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.707738 9982 init.cc:377] Logged in from keytab as oryx/127.29.230.61@KRBTEST.COM (short username oryx)
I20260810 15:16:17.707983 9982 webserver.cc:460] Webserver started at http://127.29.230.61:46417/ using document root <none> and password file <none>
I20260810 15:16:17.708091 9982 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.708123 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.708791 9982 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:17.709291 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.709398 9982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.709441 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "1966bcfcdb48443eaf4676295f979e71"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "a12044874f80d72a65b96c04e31d1e46"
server_key_iv: "09831796dbc9b2c5883187c8bf130098"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.709494 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.728011 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.729646 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.732872 9982 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.61:35991
I20260810 15:16:17.732924 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.61:35991 every 8 connection(s)
I20260810 15:16:17.733217 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
I20260810 15:16:17.733991 10294 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:16:17.734408 10294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Bootstrap starting.
I20260810 15:16:17.736793 10294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:16:17.736991 10294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: Bootstrap complete.
I20260810 15:16:17.737183 10294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } }
I20260810 15:16:17.737255 10294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1966bcfcdb48443eaf4676295f979e71, State: Initialized, Role: LEARNER
I20260810 15:16:17.737293 10294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } }
I20260810 15:16:17.737498 10294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:17.737480 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } }
I20260810 15:16:17.737573 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:16:17.738862 10317 catalog_manager.cc:1259] Loaded cluster ID: eba84c09f8ad41b6976183e903c3381c
I20260810 15:16:17.738911 10317 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: loading cluster ID for follower catalog manager: success
I20260810 15:16:17.739837 10317 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: acquiring CA information for follower catalog manager: success
I20260810 15:16:17.740114 10317 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:16:17.747974 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.742044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.29.230.61:35991)
0810 15:16:17.742158 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.742160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.742168 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:17.742190 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.742191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.742199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.742254 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.742769 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.743561 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.744266 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.744489 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.744543 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.745637 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.745638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.745639 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.745645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.746907 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.747218 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.747219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.747219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.747298 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.747469 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.747469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.747470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.747570 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.747653 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.747765 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.747847 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for oryx/127.29.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.60.keytab.
Entry for principal oryx/127.29.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.60.keytab.
I20260810 15:16:17.775473 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:36869
--webserver_interface=127.29.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.60.keytab
--principal=oryx/127.29.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:17.859983 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:17.860183 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:17.860203 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:17.861279 10329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:17.861310 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:17.861323 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:17.861334 10329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:17.861344 10329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:17.862800 10329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.60.keytab
--principal=oryx/127.29.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:36869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:17.862993 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:17.863163 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:17.867748 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.867733 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:17.867748 10342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:17.867815 10329 server_base.cc:1034] running on GCE node
I20260810 15:16:17.868167 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:17.868427 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:17.869570 10329 hybrid_clock.cc:648] HybridClock initialized: now 1786374977869552 us; error 29 us; skew 500 ppm
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:17.872004 10329 init.cc:377] Logged in from keytab as oryx/127.29.230.60@KRBTEST.COM (short username oryx)
I20260810 15:16:17.872419 10329 webserver.cc:460] Webserver started at http://127.29.230.60:46459/ using document root <none> and password file <none>
I20260810 15:16:17.872629 10329 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.872676 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.872799 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:17.873801 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/instance:
uuid: "722af23f24d4486288483b3105ef3dc1"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "efd19d0cc7a9e8170654a4db92fb8ce4"
server_key_iv: "126dbbded6d9769318d3fdc28b4e3fe3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.874131 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal/instance:
uuid: "722af23f24d4486288483b3105ef3dc1"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "efd19d0cc7a9e8170654a4db92fb8ce4"
server_key_iv: "126dbbded6d9769318d3fdc28b4e3fe3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.875394 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:17.876071 10353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.876215 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.876296 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "722af23f24d4486288483b3105ef3dc1"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "efd19d0cc7a9e8170654a4db92fb8ce4"
server_key_iv: "126dbbded6d9769318d3fdc28b4e3fe3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.876361 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.902478 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:17.904423 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:17.904557 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:17.908339 10329 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.60:36869
I20260810 15:16:17.908378 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.60:36869 every 8 connection(s)
I20260810 15:16:17.908715 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
I20260810 15:16:17.909814 10438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:17.911473 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 10329
I20260810 15:16:17.911384 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.60@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
I20260810 15:16:17.920049 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.912238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.29.230.62:37971)
0810 15:16:17.912282 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.912283 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.912337 (+ 54us) server_negotiation.cc:407] Connection header received
0810 15:16:17.912846 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.912847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.912857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.912912 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.913420 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.913941 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.914947 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.915213 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.915282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.917438 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.917440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.917440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.917447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.918660 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.919096 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.919097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.919097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.919152 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.919406 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.919408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.919408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.919549 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.919617 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.919877 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.919954 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:17.920202 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.912060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57818)
0810 15:16:17.912218 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.912261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.912749 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.912933 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.912935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.913179 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.913326 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.913331 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.914027 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.914028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.914858 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.914860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.915312 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.915313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.915431 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.915964 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.915974 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.917323 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.918735 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.918741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.918743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.919029 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.919205 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.919206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.919206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.919341 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.919616 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.919621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.919737 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.919849 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.919988 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260810 15:16:17.920717 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.912051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.29.230.60:36869)
0810 15:16:17.912292 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.912297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.912338 (+ 41us) server_negotiation.cc:407] Connection header received
0810 15:16:17.912761 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.912771 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.912797 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.912881 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.913462 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.914403 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.915141 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.915535 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.915604 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.917923 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.917930 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.917930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.917936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.919424 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.919754 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.919756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.919756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.919814 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.920018 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.920018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.920019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.920164 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.920522 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.920574 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.920649 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260810 15:16:17.920737 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.60@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:17.926455 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.920996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57900)
0810 15:16:17.921041 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.921052 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.921093 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.921203 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.921204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.921340 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.921371 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.921372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.921971 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.921971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.922460 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.922461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.922832 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.922832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.922875 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.923177 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.923180 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.924327 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.925697 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.925698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.925698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.925865 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.926030 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.926030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.926030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.926078 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.926254 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.926255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.926288 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.926331 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.926396 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:16:17.926470 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.921059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.29.230.61:35991)
0810 15:16:17.921099 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.921101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.921109 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:17.921138 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.921139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.921150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.921200 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.921433 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.921909 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.922552 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.922760 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.922821 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.924426 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.924427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.924427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.924434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.925625 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.925922 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.925922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.925923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.925966 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.926120 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.926120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.926120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.926214 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.926258 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.926348 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.926411 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:16:17.926895 10438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374977, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.60@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
I20260810 15:16:17.932698 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.927146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60626)
0810 15:16:17.927205 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.927214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.927241 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.927369 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.927369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.927516 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.927547 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.927548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.928216 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.928217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.928744 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.928746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.929129 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.929131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.929178 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.929449 (+ 271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.929452 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.930494 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.931864 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.931865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.931865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.932019 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.932174 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.932175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.932175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.932226 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.932417 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.932417 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.932453 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.932512 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.932604 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 15:16:17.932752 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.927193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.29.230.60:36869)
0810 15:16:17.927223 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.927225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.927245 (+ 20us) server_negotiation.cc:407] Connection header received
0810 15:16:17.927317 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.927317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.927326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.927368 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.927622 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.928138 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.928821 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.929067 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.929116 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.930567 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.930569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.930569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.930575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.931792 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.932076 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.932077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.932077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.932124 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.932274 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.932274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.932275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.932364 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.932418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.932541 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.932655 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":17}
I20260810 15:16:17.934108 10438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Bootstrap starting.
I20260810 15:16:17.934893 10438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:17.935148 10438 log.cc:824] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:17.935889 10438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: No bootstrap required, opened a new log
I20260810 15:16:17.937273 10438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } }
I20260810 15:16:17.937436 10438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:17.937464 10438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722af23f24d4486288483b3105ef3dc1, State: Initialized, Role: FOLLOWER
I20260810 15:16:17.937593 10438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } }
I20260810 15:16:17.937889 10438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:17.937851 10448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } } }
I20260810 15:16:17.937943 10448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:16:17.939445 10475 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:17.939499 10475 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:17.941840 10329 master_runner.cc:428] Detected that this master 722af23f24d4486288483b3105ef3dc1 is joining an existing cluster
I20260810 15:16:17.941885 10329 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.230.60:36869
I20260810 15:16:17.950062 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.942874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.29.230.62:37971)
0810 15:16:17.942922 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.942924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.943065 (+ 141us) server_negotiation.cc:407] Connection header received
0810 15:16:17.943162 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.943162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.943173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.943220 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.943617 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944389 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.945466 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.945745 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.945789 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.946922 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.946924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.946924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.946932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.948596 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949194 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.949195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.949195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.949262 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949492 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.949493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.949493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.949632 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.949695 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.949887 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.949970 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:17.950071 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.942832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57834)
0810 15:16:17.942991 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.943001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.943105 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.943222 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.943223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.943451 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.943528 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.943530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944496 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.944497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.945360 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.945363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.945877 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.945878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.945936 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.946391 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.946397 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.946825 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.948679 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.948680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.948680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.949116 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.949330 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.949332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.949429 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.949738 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.949739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.949783 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.949851 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.949951 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260810 15:16:17.950582 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.942837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57910)
0810 15:16:17.943337 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.943346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.943406 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.943662 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.943663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.943833 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.943893 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.943895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944759 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.944760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.946452 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.946456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.946944 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.946944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.946998 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.947358 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.947363 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.948057 (+ 694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.949595 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.949596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.949748 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.949928 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.949930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.949987 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.950289 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.950290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.950329 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.950394 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.950486 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":103,"threads_started":1}
I20260810 15:16:17.950618 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.943062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.29.230.61:35991)
0810 15:16:17.943102 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.943104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.943412 (+ 308us) server_negotiation.cc:407] Connection header received
0810 15:16:17.943460 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.943461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.943472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.943523 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.943979 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944678 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.946543 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.946829 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.946874 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.948133 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.948134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.948135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.948141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.949535 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.949805 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.949806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.949806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.949861 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.950067 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.950068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.950068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.950193 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.950239 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.950443 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.950528 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 15:16:17 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374976, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.62@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
I20260810 15:16:17.966470 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.952250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60642)
0810 15:16:17.952431 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.952442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.952486 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.952648 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.952649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.952788 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.952825 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.952826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.953439 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.953440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.953949 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.953950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.954283 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.954284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.954347 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.954674 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.954679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.956054 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.965408 (+ 9354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.965409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.965410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.965592 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.965785 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.965785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.965786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.965846 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.966109 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.966110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.966171 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.966246 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.966367 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 15:16:17.966470 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.952301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.29.230.60:36869)
0810 15:16:17.952462 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.952463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.952473 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:17.952571 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.952572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.952582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.952637 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.952889 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.953374 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.954015 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.954220 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.954276 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.956151 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.956152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.956152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.956158 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.965270 (+ 9112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.965661 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.965662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.965663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.965726 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.965895 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.965896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.965896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.966013 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.966104 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.966275 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.966357 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260810 15:16:17.966979 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.943175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60630)
0810 15:16:17.943484 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.943492 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.943546 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.943716 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.943716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.943912 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.943971 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.943973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944859 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.944860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.945714 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.945717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.946178 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.946178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.946236 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.963786 (+ 17550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.963791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.964160 (+ 369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.966062 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.966063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.966063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.966232 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.966404 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.966405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.966405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.966449 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.966693 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.966694 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:17.966727 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.966784 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.966882 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":69,"threads_started":1}
I20260810 15:16:17.967072 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.943113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.29.230.60:36869)
0810 15:16:17.943149 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.943150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.943529 (+ 379us) server_negotiation.cc:407] Connection header received
0810 15:16:17.943637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.943638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.943645 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.943669 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.944062 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.944776 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.945800 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.946076 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.946114 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.964301 (+ 18187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.964303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.964303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.964311 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.965975 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.966302 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.966303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.966303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.966348 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.966505 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.966505 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.966505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.966614 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.966653 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.966833 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.966947 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":16}
I20260810 15:16:17.967425 9085 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.230.60:36869: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "722af23f24d4486288483b3105ef3dc1"
member_type: NON_VOTER
last_known_addr {
host: "127.29.230.60"
port: 36869
}
attrs {
promote: true
}
}
dest_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42"
cas_config_opid_index: 5
I20260810 15:16:17.967697 9085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } }
I20260810 15:16:17.968118 10158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 722af23f24d4486288483b3105ef3dc1 (127.29.230.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.968923 10158 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.230.60:36869
I20260810 15:16:17.969000 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.969084 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:17.969386 10329 master.cc:537] Master@127.29.230.60:36869 shutting down...
I20260810 15:16:17.969537 10397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:17.970355 10397 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 15:16:17.970624 10157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:16:17.971365 9061 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 -> Peer 722af23f24d4486288483b3105ef3dc1 (127.29.230.60:36869): Couldn't send request to peer 722af23f24d4486288483b3105ef3dc1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 15:16:17.973536 10329 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:17.973631 10329 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:17.973682 10329 tablet_replica.cc:330] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: stopping tablet replica
I20260810 15:16:17.975047 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.968167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.29.230.61:35991)
0810 15:16:17.968209 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.968211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.968219 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:17.968245 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.968246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.968272 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.968347 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:17.968502 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.969277 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.970337 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.973935 (+ 3598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.974007 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.974095 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:17.974934 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 15:16:17.974979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:16:17.974390 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.968106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:57912)
0810 15:16:17.968147 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.968156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.968187 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.968369 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.968370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.968372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:17.968419 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.968420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.969347 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.969348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.970246 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.970249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.974131 (+ 3882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.974133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.974205 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.974227 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 15:16:17.974284 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:16:17.974326 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:16:17.975518 10242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 LEARNER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 15:16:17.975729 10157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 15:16:17.976429 10242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 722af23f24d4486288483b3105ef3dc1 (127.29.230.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.977523 10492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.977613 10492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:16:17.977746 10157 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: attempting to promote NON_VOTER 1966bcfcdb48443eaf4676295f979e71 to VOTER
I20260810 15:16:17.977753 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3b0d6e183f4d08b5ad1f64f18b9e42. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: NON_VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: true } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.977813 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:16:17.977934 10157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } }
I20260810 15:16:17.978348 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.978259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60646)
0810 15:16:17.978288 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.978327 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260810 15:16:17.978426 10242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 LEARNER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 15:16:17.978605 10158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 15:16:17.978794 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.978714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60652)
0810 15:16:17.978743 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.978771 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 15:16:17.978955 9061 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 -> Peer 722af23f24d4486288483b3105ef3dc1 (127.29.230.60:36869): Couldn't send request to peer 722af23f24d4486288483b3105ef3dc1. Status: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36869: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 15:16:17.979203 10159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.979383 10242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.980011 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.980103 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:17.980149 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.980188 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:17.980232 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:17.980309 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:16:17.986776 10329 master.cc:552] Master@127.29.230.60:36869 shutdown complete.
I20260810 15:16:17.986820 10329 master_runner.cc:305] Clearing existing system tablet
I20260810 15:16:17.986934 10329 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:17.986975 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:17.987644 10329 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:17.988065 10494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.988211 10329 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:17.988266 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:17.988286 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "722af23f24d4486288483b3105ef3dc1"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "efd19d0cc7a9e8170654a4db92fb8ce4"
server_key_iv: "126dbbded6d9769318d3fdc28b4e3fe3"
server_key_version: "encryptionkey@0"
I20260810 15:16:17.988894 10329 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:17.989955 10329 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 15:16:17.990008 10329 log.cc:1196] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 15:16:17.990168 10329 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Deleting consensus metadata
I20260810 15:16:17.990272 10329 master_runner.cc:315] Copying system tablet from 127.29.230.62:37971
I20260810 15:16:17.990779 10329 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.230.62:37971
I20260810 15:16:18.000288 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.991109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.29.230.62:37971)
0810 15:16:17.991159 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:17.991161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:17.994826 (+ 3665us) server_negotiation.cc:407] Connection header received
0810 15:16:17.994885 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:17.994886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:17.994897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:17.994945 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:17.995297 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.995819 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.996564 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.996764 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.996816 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.997985 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:17.997987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:17.997987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:17.997995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:17.999241 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.999601 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.999602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.999602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.999659 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:17.999816 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:17.999816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:17.999817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:17.999926 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:17.999980 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.000124 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.000198 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:16:18.000293 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:17.991058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:57848)
0810 15:16:17.994748 (+ 3690us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:17.994759 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:17.994824 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:17.994947 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:17.994948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:17.995149 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:17.995206 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.995209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.995902 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.995903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.996481 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:17.996483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:17.996822 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:17.996823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:17.996888 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:17.997614 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:17.997619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:17.997901 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:17.999365 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.999366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.999367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.999533 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.999710 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:17.999711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:17.999711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:17.999767 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:17.999979 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:17.999979 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.000025 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.000094 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.000199 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3669,"thread_start_us":71,"threads_started":1}
I20260810 15:16:18.000703 9129 tablet_copy_service.cc:140] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 722af23f24d4486288483b3105ef3dc1 ({username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:57848)
I20260810 15:16:18.000759 9129 tablet_copy_service.cc:161] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 722af23f24d4486288483b3105ef3dc1 at {username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:57848: session id = 722af23f24d4486288483b3105ef3dc1-00000000000000000000000000000000
I20260810 15:16:18.001183 9129 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 15:16:18.001622 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 15:16:18.002519 10329 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: tablet copy: Starting download of 0 data blocks...
I20260810 15:16:18.002643 10329 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: tablet copy: Starting download of 1 WAL segments...
I20260810 15:16:18.003360 10329 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 15:16:18.004112 9129 tablet_copy_service.cc:342] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Request end of tablet copy session 722af23f24d4486288483b3105ef3dc1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:57848
I20260810 15:16:18.004166 9129 tablet_copy_service.cc:433] P 2a3b0d6e183f4d08b5ad1f64f18b9e42: ending tablet copy session 722af23f24d4486288483b3105ef3dc1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 722af23f24d4486288483b3105ef3dc1
I20260810 15:16:18.006453 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:18.007609 10506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.007716 10508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:18.007822 10329 server_base.cc:1034] running on GCE node
W20260810 15:16:18.007918 10512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:18.008065 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:18.008203 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:18.009336 10329 hybrid_clock.cc:648] HybridClock initialized: now 1786374978009321 us; error 28 us; skew 500 ppm
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:18.011127 10329 init.cc:377] Logged in from keytab as oryx/127.29.230.60@KRBTEST.COM (short username oryx)
I20260810 15:16:18.011430 10329 webserver.cc:460] Webserver started at http://127.29.230.60:45559/ using document root <none> and password file <none>
I20260810 15:16:18.011559 10329 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:18.011591 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:18.012147 10329 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:18.012629 10522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.012758 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.012818 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "722af23f24d4486288483b3105ef3dc1"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "efd19d0cc7a9e8170654a4db92fb8ce4"
server_key_iv: "126dbbded6d9769318d3fdc28b4e3fe3"
server_key_version: "encryptionkey@0"
I20260810 15:16:18.012880 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:18.030155 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:18.031823 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:18.035045 10329 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.60:36869
I20260810 15:16:18.035109 10625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.60:36869 every 8 connection(s)
I20260810 15:16:18.035415 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
I20260810 15:16:18.036110 10632 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:16:18.036509 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Bootstrap starting.
I20260810 15:16:18.038879 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:16:18.039057 10632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Bootstrap complete.
I20260810 15:16:18.039220 10632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } }
I20260810 15:16:18.039321 10632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 722af23f24d4486288483b3105ef3dc1, State: Initialized, Role: LEARNER
I20260810 15:16:18.039354 10632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } }
I20260810 15:16:18.039568 10632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:18.039567 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:18.039659 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: LEARNER
I20260810 15:16:18.040920 10657 catalog_manager.cc:1259] Loaded cluster ID: eba84c09f8ad41b6976183e903c3381c
I20260810 15:16:18.040967 10657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: loading cluster ID for follower catalog manager: success
I20260810 15:16:18.041973 10657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: acquiring CA information for follower catalog manager: success
I20260810 15:16:18.042263 10657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:16:18.083563 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.076580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.29.230.60:36869)
0810 15:16:18.076700 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.076702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.076708 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:16:18.076730 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.076731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.076741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.076798 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.077527 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.078214 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.079213 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.079461 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.079581 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.081061 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.081062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.081063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.081070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.082328 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.082727 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.082728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.082728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.082784 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.082968 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.082969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.082969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.083094 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.083169 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.083385 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.083465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260810 15:16:18.084088 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 9217
I20260810 15:16:18.097716 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:44389
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:37971
--webserver_port=34773
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:18.179759 10661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:18.179967 10661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:18.179991 10661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:18.181202 10661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:18.181245 10661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:18.181270 10661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:18.182801 10661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:44389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=34773
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:18.183024 10661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:18.183256 10661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:18.188314 10676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.188321 10669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.188304 10670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:18.188431 10661 server_base.cc:1034] running on GCE node
I20260810 15:16:18.188726 10661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:18.188946 10661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:18.190104 10661 hybrid_clock.cc:648] HybridClock initialized: now 1786374978190086 us; error 31 us; skew 500 ppm
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:18.192638 10661 init.cc:377] Logged in from keytab as oryx/127.29.230.1@KRBTEST.COM (short username oryx)
I20260810 15:16:18.193042 10661 webserver.cc:460] Webserver started at http://127.29.230.1:34773/ using document root <none> and password file <none>
I20260810 15:16:18.193243 10661 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:18.193307 10661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:18.194645 10661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:18.195238 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.195407 10661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:18.195490 10661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "8359f4ef0b1d4e9083872a01426bc7cf"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "aa6fb1b4711f4763e369d546202aafec"
server_key_iv: "2fb04bdeb58c68be0491394c8c43d2a3"
server_key_version: "encryptionkey@0"
I20260810 15:16:18.195843 10661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:18.211606 10661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:18.213634 10661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:18.213768 10661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:18.213990 10661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:18.214287 10661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:18.214319 10661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.214347 10661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:18.214371 10661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.220512 10661 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44389
I20260810 15:16:18.220571 10909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44389 every 8 connection(s)
I20260810 15:16:18.220899 10661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:18.223048 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 10661
I20260810 15:16:18.223212 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 9484
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:18.242089 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:46197
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:37971
--webserver_port=36923
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:16:18.242511 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.222058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:50307 (local address 127.29.230.61:35991)
0810 15:16:18.222115 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.222117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.222125 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:18.222954 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.222955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.222968 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.223001 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.224319 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.225015 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.229004 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.229241 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.229301 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.239235 (+ 9934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.239236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.239237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.239245 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.240865 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241421 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241484 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241726 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241875 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.241942 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.242345 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.242427 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:16:18.242542 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.221884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:55749 (local address 127.29.230.60:36869)
0810 15:16:18.221941 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.221943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.221954 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:18.222957 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.222958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.222971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.223001 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.224103 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.224882 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226329 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.226717 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226768 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.239149 (+ 12381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.239151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.239151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.239161 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.240771 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241362 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241421 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241727 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241865 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.241924 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.242348 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.242443 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:16:18.242712 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.221621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:36673)
0810 15:16:18.221874 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.221886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.222844 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.223232 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.223233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.223561 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.223797 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.223802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.225024 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.225025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.226321 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.226324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.226731 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.226852 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.227463 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.227467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.238225 (+ 10758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.241116 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.241117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241378 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241592 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241592 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.241593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241658 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241967 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.241975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.242171 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.242300 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.242479 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":126,"thread_start_us":101,"threads_started":1}
I20260810 15:16:18.242861 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.221948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.29.230.1:50307)
0810 15:16:18.222072 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.222081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.222846 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.223149 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.223154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.224171 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.224221 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.224223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.225121 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.225122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.228914 (+ 3792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.228917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.229326 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.229328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.229396 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.229774 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.229779 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.239112 (+ 9333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.240992 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.240994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.240994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241328 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241558 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.241559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241657 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241978 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.241979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.242170 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.242600 (+ 430us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.242768 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":127,"thread_start_us":84,"threads_started":1}
I20260810 15:16:18.243297 10914 heartbeater.cc:347] Connected to a master server at 127.29.230.61:35991
I20260810 15:16:18.243377 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.221620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.29.230.1:55749)
0810 15:16:18.221811 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.221853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.222824 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.223149 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.223154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.223968 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.224015 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.224016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.225004 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.225005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.226215 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.226218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.226857 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.226931 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.227418 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.227430 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.239018 (+ 11588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.240883 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.240895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.240899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241273 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241492 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.241493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.241648 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.241969 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.241975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.242171 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.242701 (+ 530us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.243212 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":277,"thread_start_us":92,"threads_started":1}
I20260810 15:16:18.243419 10914 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.243635 10914 heartbeater.cc:511] Master 127.29.230.61:35991 requested a full tablet report, sending...
I20260810 15:16:18.243901 10910 heartbeater.cc:347] Connected to a master server at 127.29.230.60:36869
I20260810 15:16:18.243952 10910 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.244010 10910 heartbeater.cc:511] Master 127.29.230.60:36869 requested a full tablet report, sending...
I20260810 15:16:18.244119 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.221617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:36673 (local address 127.29.230.62:37971)
0810 15:16:18.221673 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.221675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.221991 (+ 316us) server_negotiation.cc:407] Connection header received
0810 15:16:18.223059 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.223061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.223073 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.223126 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.223902 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.224912 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226407 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.226630 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.226673 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.238335 (+ 11662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.238337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.238338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.238348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.240998 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241458 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241517 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.241722 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.241723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.241724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.241865 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.241939 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.243931 (+ 1992us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.244038 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:16:18.244215 10225 ts_manager.cc:194] Registered new tserver with Master: 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1:44389)
I20260810 15:16:18.244383 10554 ts_manager.cc:194] Registered new tserver with Master: 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1:44389)
I20260810 15:16:18.244510 10912 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:18.244565 10912 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.244624 10912 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:18.244906 9085 ts_manager.cc:194] Re-registered known tserver with Master: 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1:44389)
I20260810 15:16:18.245281 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:36673
W20260810 15:16:18.327255 10920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:18.327492 10920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:18.327525 10920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:18.328675 10920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:18.328717 10920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:18.328752 10920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:18.330214 10920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:46197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=36923
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:18.330451 10920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:18.330678 10920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:18.335872 10928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.335878 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.335878 10927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:18.335940 10920 server_base.cc:1034] running on GCE node
I20260810 15:16:18.336218 10920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:18.336465 10920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:18.337621 10920 hybrid_clock.cc:648] HybridClock initialized: now 1786374978337600 us; error 37 us; skew 500 ppm
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:18.340412 10920 init.cc:377] Logged in from keytab as oryx/127.29.230.2@KRBTEST.COM (short username oryx)
I20260810 15:16:18.340875 10920 webserver.cc:460] Webserver started at http://127.29.230.2:36923/ using document root <none> and password file <none>
I20260810 15:16:18.341094 10920 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:18.341159 10920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:18.342576 10920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:18.343252 10942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.343456 10920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:18.343554 10920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "2317b4eb6d794b208ca390c7924312cb"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "925f068a4142c8c78a18875ca11b4569"
server_key_iv: "b8e59167012c1ef9418e0473174f4203"
server_key_version: "encryptionkey@0"
I20260810 15:16:18.343940 10920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:18.358712 10920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:18.360724 10920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:18.360867 10920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:18.361120 10920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:18.361428 10920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:18.361461 10920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.361492 10920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:18.361514 10920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.367305 10920 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:46197
I20260810 15:16:18.367359 11138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:46197 every 8 connection(s)
I20260810 15:16:18.367695 10920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:18.369050 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 10920
I20260810 15:16:18.369200 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 9730
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:18.383015 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:41137 (local address 127.29.230.61:35991)
0810 15:16:18.368681 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.368684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.368892 (+ 208us) server_negotiation.cc:407] Connection header received
0810 15:16:18.372084 (+ 3192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.372086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.372104 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.372149 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.373769 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.374390 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.376320 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.376697 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.376748 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.379685 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.379687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.379688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.379697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.381428 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.381907 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.381908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.381908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.381983 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.382249 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.382250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.382251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.382433 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.382515 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.382801 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.382904 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260810 15:16:18.383100 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.29.230.2:41137)
0810 15:16:18.368613 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.368663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.371981 (+ 3318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.372282 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.372288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.373272 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.373666 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.373677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.374493 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.374494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.376191 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.376195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.376857 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.376859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.377040 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.377743 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.377756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.379577 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.381511 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.381517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.381520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.381799 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.382055 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.382055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.382056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.382174 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.382498 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.382501 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.382652 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.382732 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.382887 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260810 15:16:18.383963 11142 heartbeater.cc:347] Connected to a master server at 127.29.230.61:35991
I20260810 15:16:18.384150 11142 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.384482 11142 heartbeater.cc:511] Master 127.29.230.61:35991 requested a full tablet report, sending...
I20260810 15:16:18.385061 10225 ts_manager.cc:194] Registered new tserver with Master: 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:18.390489 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:32875 (local address 127.29.230.62:37971)
0810 15:16:18.368910 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.368912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.383368 (+ 14456us) server_negotiation.cc:407] Connection header received
0810 15:16:18.383451 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.383452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.383464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.383489 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.383902 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.384510 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.385283 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.385504 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.385534 (+ 30us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.387870 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.387873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.387873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.387881 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.389289 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.389730 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.389730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.389731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.389780 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.389983 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.389984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.389984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.390096 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.390145 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.390335 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.390410 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:16:18.390556 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:32875)
0810 15:16:18.383322 (+ 14557us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.383331 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.383394 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.383541 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.383541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.383669 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.383721 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.383723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.384586 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.384587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.385209 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.385211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.385603 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.385603 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.385665 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.386106 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.386110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.387771 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.389380 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.389382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.389382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.389650 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.389841 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.389842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.389843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.389921 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.390164 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.390165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.390229 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.390320 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.390449 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14545,"thread_start_us":71,"threads_started":1}
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
I20260810 15:16:18.404105 11141 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:18.404196 11141 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.404289 11141 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:18.404778 9085 ts_manager.cc:194] Re-registered known tserver with Master: 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
I20260810 15:16:18.405277 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:32875
I20260810 15:16:18.406791 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:36357 (local address 127.29.230.60:36869)
0810 15:16:18.368499 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.368502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.368759 (+ 257us) server_negotiation.cc:407] Connection header received
0810 15:16:18.391707 (+ 22948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.391709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.391726 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.391784 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.392274 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.393231 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.394509 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.394890 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.394943 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.398333 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.398334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.398334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.398342 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.405732 (+ 7390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.406139 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.406141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.406141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.406203 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.406378 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.406378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.406378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.406474 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.406518 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.406660 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.406728 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:18.406880 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.368433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.29.230.2:36357)
0810 15:16:18.368613 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.368663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.369256 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.391848 (+ 22592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.391849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.392055 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.392152 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.392155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.393341 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.393342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.394387 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.394391 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.395034 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.395035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.395113 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.396343 (+ 1230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.396348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.398224 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.405815 (+ 7591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.405816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.405817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.406044 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.406262 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.406263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.406263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.406326 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.406519 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.406520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.406570 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.406635 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.406764 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260810 15:16:18.407377 11139 heartbeater.cc:347] Connected to a master server at 127.29.230.60:36869
I20260810 15:16:18.407444 11139 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.407521 11139 heartbeater.cc:511] Master 127.29.230.60:36869 requested a full tablet report, sending...
I20260810 15:16:18.407827 10554 ts_manager.cc:194] Registered new tserver with Master: 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
I20260810 15:16:18.409181 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:33353
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:37971
--webserver_port=36303
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:16:18.488834 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.486183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:60682)
0810 15:16:18.486275 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.486286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.486346 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.486519 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.486520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.486523 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.486583 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.486585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.487237 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.487238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.488031 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.488034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.488583 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.488585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.488649 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.488653 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.488708 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.488755 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":40,"threads_started":1}
I20260810 15:16:18.488830 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.486295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.29.230.60:36869)
0810 15:16:18.486360 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.486362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.486372 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:18.486401 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.486402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.486426 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.486503 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.486639 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.487154 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.488114 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.488508 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.488569 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.488650 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.488716 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.488752 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:16:18.489142 10572 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 15:16:18.489341 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a3b0d6e183f4d08b5ad1f64f18b9e42. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:18.489456 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: LEARNER
W20260810 15:16:18.492928 11148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:18.493129 11148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:18.493152 11148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:18.494311 11148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:18.494352 11148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:18.494378 11148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:18.495915 11148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:33353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=36303
--tserver_master_addrs=127.29.230.62:37971,127.29.230.61:35991,127.29.230.60:36869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:18.496160 11148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:18.496392 11148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:18.501459 11164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.501477 11162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:18.501469 11159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:18.502031 11148 server_base.cc:1034] running on GCE node
I20260810 15:16:18.502247 11148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:18.502476 11148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:18.503636 11148 hybrid_clock.cc:648] HybridClock initialized: now 1786374978503616 us; error 27 us; skew 500 ppm
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:18.506270 11148 init.cc:377] Logged in from keytab as oryx/127.29.230.3@KRBTEST.COM (short username oryx)
I20260810 15:16:18.506680 11148 webserver.cc:460] Webserver started at http://127.29.230.3:36303/ using document root <none> and password file <none>
I20260810 15:16:18.506884 11148 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:18.506937 11148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:18.508351 11148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:18.509100 11174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.509276 11148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:18.509361 11148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "f8022693fa844a238473161008ed0b49"
format_stamp: "Formatted at 2026-08-10 15:16:17 on dist-test-slave-gjrz"
server_key: "62305ffdccc5d88767f2406eee72b1d0"
server_key_iv: "ab79e1787d3ce8aa76b2e9f09824d95b"
server_key_version: "encryptionkey@0"
I20260810 15:16:18.509783 11148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:18.521153 11148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:18.523161 11148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:18.523316 11148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:18.523576 11148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:18.523903 11148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:18.523937 11148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.523958 11148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:18.523972 11148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:18.529553 11148 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:33353
I20260810 15:16:18.529597 11366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:33353 every 8 connection(s)
I20260810 15:16:18.529923 11148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:18.534879 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 11148
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.60@KRBTEST.COM
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
Aug 10 15:16:18 dist-test-slave-gjrz krb5kdc[9000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374978, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.61@KRBTEST.COM
I20260810 15:16:18.542891 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:34629 (local address 127.29.230.60:36869)
0810 15:16:18.531009 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.531011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.531019 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:18.532089 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.532090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.532103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.532167 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.533013 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.533808 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535178 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.535502 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535573 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.539629 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.539630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.539630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.539637 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.541143 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.541611 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.541612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.541613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.541672 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.541954 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.541955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.541956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.542106 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.542171 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.542717 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.542798 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:16:18.543439 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.29.230.3:34629)
0810 15:16:18.530890 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.530916 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.531993 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.532253 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.532257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.532714 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.532924 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.532929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.533935 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.533936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.535087 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.535090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.535657 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.535801 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.536522 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.536526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.539530 (+ 3004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.541253 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.541254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.541255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.541542 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.541722 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.541723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.541723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.541854 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.542359 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.542366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.542540 (+ 174us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.542992 (+ 452us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.543184 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260810 15:16:18.544082 11367 heartbeater.cc:347] Connected to a master server at 127.29.230.60:36869
I20260810 15:16:18.544181 11367 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.544380 11367 heartbeater.cc:511] Master 127.29.230.60:36869 requested a full tablet report, sending...
I20260810 15:16:18.544842 10554 ts_manager.cc:194] Registered new tserver with Master: f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:18.545966 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:55307)
0810 15:16:18.531118 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.531127 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.532005 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.532256 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.532258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.532695 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.532925 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.532932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.534468 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.534469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.535256 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.535258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.535798 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.535862 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.536468 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.536478 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.538899 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.541117 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.541124 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.541127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.541676 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.545237 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.545239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.545308 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.545628 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.545629 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.545671 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.545734 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.545846 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":73,"thread_start_us":89,"threads_started":1}
I20260810 15:16:18.546205 10319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:35173 (local address 127.29.230.61:35991)
0810 15:16:18.531013 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.531015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.531024 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:18.532135 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.532136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.532147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.532177 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.533050 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.533808 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.536086 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.536382 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.536452 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.539524 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.539525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.539525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.539534 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.543317 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545270 (+ 1953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.545272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.545272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.545327 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545532 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.545533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.545533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.545669 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.545730 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.546034 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.546111 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:16:18.546237 11375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.29.230.3:35173)
0810 15:16:18.530874 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.530916 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.531990 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.532466 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.532467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.532706 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:18.532925 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.532932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.535181 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.535963 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.535965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.536451 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.536452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.536554 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.536926 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:18.536931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:18.539369 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:18.544509 (+ 5140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.544510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.544511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.544704 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.545394 (+ 690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:18.545395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:18.545450 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:18.545761 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:18.545762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:18.545810 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.545865 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.545962 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260810 15:16:18.546293 11387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.536248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.29.230.62:37971)
0810 15:16:18.536831 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.536833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.536841 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:18.536862 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.536863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.536875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.537555 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.538342 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.539708 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.540803 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.541091 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.541407 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.542661 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.542663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.542663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.542669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.544326 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.544918 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.544919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.544919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.544977 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545195 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.545196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.545196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.545698 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.546084 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.546125 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.546209 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":54,"threads_started":1}
I20260810 15:16:18.546546 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.530723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:55307 (local address 127.29.230.62:37971)
0810 15:16:18.530792 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.530794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.531188 (+ 394us) server_negotiation.cc:407] Connection header received
0810 15:16:18.532124 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.532126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.532139 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.532171 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.533914 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.534381 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535366 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.535680 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.535722 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.538980 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.538981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.538982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.538990 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.541027 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.542327 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.542328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.542328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.542393 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.545384 (+ 2991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.545385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.545385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.545533 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.545598 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.546437 (+ 839us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.546497 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:16:18.547155 11370 heartbeater.cc:347] Connected to a master server at 127.29.230.62:37971
I20260810 15:16:18.547214 11370 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.547309 11370 heartbeater.cc:511] Master 127.29.230.62:37971 requested a full tablet report, sending...
I20260810 15:16:18.547485 11371 heartbeater.cc:347] Connected to a master server at 127.29.230.61:35991
I20260810 15:16:18.547523 11371 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:18.547577 11371 heartbeater.cc:511] Master 127.29.230.61:35991 requested a full tablet report, sending...
I20260810 15:16:18.547722 9085 ts_manager.cc:194] Re-registered known tserver with Master: f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:18.547847 10225 ts_manager.cc:194] Registered new tserver with Master: f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:18.548168 9085 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:55307
I20260810 15:16:18.548478 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.536854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.29.230.60:36869)
0810 15:16:18.537624 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.537626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.538016 (+ 390us) server_negotiation.cc:407] Connection header received
0810 15:16:18.538084 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.538085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.538098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.538157 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.538761 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.540474 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.541779 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.542091 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.542154 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.544192 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.544193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.544193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.544200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.546557 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.546901 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.546903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.546903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.546971 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.547188 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.547189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.547189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.547995 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.548105 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.548292 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.548375 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":62,"threads_started":1}
I20260810 15:16:18.550709 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.537246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.29.230.61:35991)
0810 15:16:18.537671 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.537672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.537779 (+ 107us) server_negotiation.cc:407] Connection header received
0810 15:16:18.537822 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.537823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.537836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.537891 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.538731 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.543713 (+ 4982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.544627 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.544919 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.544973 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.548179 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.548180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.548180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.548187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.549776 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.550069 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.550070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.550071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.550132 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.550265 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.550265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.550266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.550385 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.550470 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.550545 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.550625 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":55,"threads_started":1}
I20260810 15:16:18.548804 9085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:18.550925 9085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:18.551693 11149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: NON_VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: true } } }
I20260810 15:16:18.551788 11149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:18.552856 11398 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: attempting to promote NON_VOTER 722af23f24d4486288483b3105ef3dc1 to VOTER
I20260810 15:16:18.552997 11398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } }
I20260810 15:16:18.553393 10242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 15:16:18.553439 10572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 LEARNER]: Refusing update from remote peer 2a3b0d6e183f4d08b5ad1f64f18b9e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 15:16:18.553574 11149 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 15:16:18.553689 11399 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 15:16:18.554162 11407 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 722af23f24d4486288483b3105ef3dc1 (127.29.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.554298 10242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 722af23f24d4486288483b3105ef3dc1 (127.29.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.554622 10572 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 722af23f24d4486288483b3105ef3dc1 (127.29.230.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555078 11407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555145 11407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:18.555186 11407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555225 11407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:18.555238 11149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555342 11149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:18.555533 11414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555616 11414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:16:18.555658 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:18.555768 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:16:18.560086 11151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:46197 (local address 127.0.0.1:50568)
0810 15:16:18.556443 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.556454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.556494 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.556777 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.556778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.556780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.556823 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.556825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.557950 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.557951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.558941 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.558944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.559855 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.559856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.559920 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.559924 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.559949 (+ 25us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.559992 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 15:16:18.560621 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.29.230.2:46197)
0810 15:16:18.556628 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.556634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.556644 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:18.556683 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.556685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.556699 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.556771 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.556876 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.557857 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.559047 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.559764 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.560394 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.560453 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.560480 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.560520 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260810 15:16:18.560774 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:33353 (local address 127.0.0.1:47654)
0810 15:16:18.557166 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.557174 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.557223 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.557387 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.557388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.557389 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.557421 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.557422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.558677 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.558678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.559764 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.559766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.560521 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.560522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.560586 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.560590 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.560636 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.560677 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":62,"threads_started":1}
I20260810 15:16:18.560684 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44389 (local address 127.0.0.1:58684)
0810 15:16:18.557014 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.557023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.557078 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.557314 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.557314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.557316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.557354 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.557355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.558334 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.558335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.559470 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.559473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.560464 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.560466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.560526 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.560530 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.560577 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.560615 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":81,"threads_started":1}
I20260810 15:16:18.561015 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.29.230.3:33353)
0810 15:16:18.556730 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.556735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.557240 (+ 505us) server_negotiation.cc:407] Connection header received
0810 15:16:18.557284 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.557286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.557302 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.557381 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.557473 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.558585 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.559833 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.560448 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.560515 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.560865 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.560895 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.560938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260810 15:16:18.561475 11023 tablet_service.cc:1467] Processing CreateTablet for tablet a77d1f9607454488a334af40b8eb5810 (DEFAULT_TABLE table=test-table [id=ff4e7e455a204c7aac3a95f26b2058cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:18.561614 11250 tablet_service.cc:1467] Processing CreateTablet for tablet a77d1f9607454488a334af40b8eb5810 (DEFAULT_TABLE table=test-table [id=ff4e7e455a204c7aac3a95f26b2058cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:18.561866 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77d1f9607454488a334af40b8eb5810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:18.561906 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77d1f9607454488a334af40b8eb5810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:18.563361 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.556744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.29.230.1:44389)
0810 15:16:18.556902 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.556906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.557085 (+ 179us) server_negotiation.cc:407] Connection header received
0810 15:16:18.557150 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.557153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.557169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.557244 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.557429 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.558263 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.559575 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.560290 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.560348 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.560428 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.560729 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.560770 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260810 15:16:18.563985 10777 tablet_service.cc:1467] Processing CreateTablet for tablet a77d1f9607454488a334af40b8eb5810 (DEFAULT_TABLE table=test-table [id=ff4e7e455a204c7aac3a95f26b2058cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:18.564318 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77d1f9607454488a334af40b8eb5810. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:18.565476 11427 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Bootstrap starting.
I20260810 15:16:18.566159 11427 tablet_bootstrap.cc:654] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:18.566514 11427 log.cc:824] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:18.566877 11428 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Bootstrap starting.
I20260810 15:16:18.567318 11427 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: No bootstrap required, opened a new log
I20260810 15:16:18.567387 11427 ts_tablet_manager.cc:1397] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:18.567739 11428 tablet_bootstrap.cc:654] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:18.568027 11428 log.cc:824] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:18.568348 11430 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Bootstrap starting.
I20260810 15:16:18.569010 11428 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: No bootstrap required, opened a new log
I20260810 15:16:18.569072 11428 ts_tablet_manager.cc:1397] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:16:18.569047 11427 raft_consensus.cc:348] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.569387 11427 raft_consensus.cc:374] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:18.569430 11427 raft_consensus.cc:729] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8022693fa844a238473161008ed0b49, State: Initialized, Role: FOLLOWER
I20260810 15:16:18.569558 11427 consensus_queue.cc:260] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.569842 11427 ts_tablet_manager.cc:1428] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:18.570065 11370 heartbeater.cc:503] Master 127.29.230.62:37971 was elected leader, sending a full tablet report...
I20260810 15:16:18.571308 11428 raft_consensus.cc:348] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.571452 11428 raft_consensus.cc:374] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:18.571502 11428 raft_consensus.cc:729] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2317b4eb6d794b208ca390c7924312cb, State: Initialized, Role: FOLLOWER
I20260810 15:16:18.571617 11428 consensus_queue.cc:260] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.571779 11430 tablet_bootstrap.cc:654] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:18.571864 11428 ts_tablet_manager.cc:1428] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 15:16:18.571945 11141 heartbeater.cc:503] Master 127.29.230.62:37971 was elected leader, sending a full tablet report...
I20260810 15:16:18.572080 11430 log.cc:824] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:18.573312 11430 tablet_bootstrap.cc:492] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: No bootstrap required, opened a new log
I20260810 15:16:18.573378 11430 ts_tablet_manager.cc:1397] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 15:16:18.575110 11430 raft_consensus.cc:348] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.575239 11430 raft_consensus.cc:374] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:18.575289 11430 raft_consensus.cc:729] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8359f4ef0b1d4e9083872a01426bc7cf, State: Initialized, Role: FOLLOWER
I20260810 15:16:18.575414 11430 consensus_queue.cc:260] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.575657 11430 ts_tablet_manager.cc:1428] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:16:18.575750 10912 heartbeater.cc:503] Master 127.29.230.62:37971 was elected leader, sending a full tablet report...
W20260810 15:16:18.654886 11143 tablet.cc:2369] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:16:18.721913 10915 tablet.cc:2369] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:18.726079 11438 raft_consensus.cc:482] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:18.726226 11438 raft_consensus.cc:504] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.726562 11438 leader_election.cc:288] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197), f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:18.729359 11422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.726875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:59415 (local address 127.29.230.3:33353)
0810 15:16:18.726943 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.726945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.727035 (+ 90us) server_negotiation.cc:407] Connection header received
0810 15:16:18.727092 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.727093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.727103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.727184 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.727348 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.727815 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.728674 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.729029 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.729093 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.729102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.729248 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.729287 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:16:18.729365 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.726910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:46917 (local address 127.29.230.2:46197)
0810 15:16:18.726958 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.726960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.726969 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:18.727000 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.727001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.727010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.727062 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:18.727211 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.727722 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.728659 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.729030 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.729094 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.729102 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.729256 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.729295 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:16:18.729405 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.726792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:33353 (local address 127.29.230.1:59415)
0810 15:16:18.726926 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.726937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.727036 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.727197 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.727198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.727200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.727291 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.727292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.727890 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.727892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.728586 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.728588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.729104 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.729105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.729185 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.729188 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.729233 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.729281 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":83}
I20260810 15:16:18.729419 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.726792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:46197 (local address 127.29.230.1:46917)
0810 15:16:18.726843 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:18.726858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:18.726936 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:18.727072 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:18.727073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:18.727074 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:18.727147 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.727149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.727803 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.727804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.728575 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:18.728577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.729107 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.729108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:18.729196 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.729199 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:18.729244 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:18.729289 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":3}
I20260810 15:16:18.729943 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d1f9607454488a334af40b8eb5810" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2317b4eb6d794b208ca390c7924312cb" is_pre_election: true
I20260810 15:16:18.729996 11285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d1f9607454488a334af40b8eb5810" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8022693fa844a238473161008ed0b49" is_pre_election: true
I20260810 15:16:18.730105 11061 raft_consensus.cc:2454] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 0.
I20260810 15:16:18.730134 11285 raft_consensus.cc:2454] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 0.
I20260810 15:16:18.730438 10688 leader_election.cc:302] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2317b4eb6d794b208ca390c7924312cb, 8359f4ef0b1d4e9083872a01426bc7cf; no voters:
I20260810 15:16:18.730574 11438 raft_consensus.cc:2781] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:18.730638 11438 raft_consensus.cc:482] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:18.730662 11438 raft_consensus.cc:3037] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:18.731478 11438 raft_consensus.cc:504] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.731632 11438 leader_election.cc:288] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 election: Requested vote from peers 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197), f8022693fa844a238473161008ed0b49 (127.29.230.3:33353)
I20260810 15:16:18.731784 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d1f9607454488a334af40b8eb5810" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2317b4eb6d794b208ca390c7924312cb"
I20260810 15:16:18.731833 11285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77d1f9607454488a334af40b8eb5810" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8022693fa844a238473161008ed0b49"
I20260810 15:16:18.731870 11061 raft_consensus.cc:3037] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:18.731907 11285 raft_consensus.cc:3037] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:18.732637 11285 raft_consensus.cc:2454] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 1.
I20260810 15:16:18.732641 11061 raft_consensus.cc:2454] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 1.
I20260810 15:16:18.732846 10686 leader_election.cc:302] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8359f4ef0b1d4e9083872a01426bc7cf, f8022693fa844a238473161008ed0b49; no voters:
I20260810 15:16:18.732960 11438 raft_consensus.cc:2781] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:18.733093 11438 raft_consensus.cc:686] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 LEADER]: Becoming Leader. State: Replica: 8359f4ef0b1d4e9083872a01426bc7cf, State: Running, Role: LEADER
I20260810 15:16:18.733202 11438 consensus_queue.cc:237] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } }
I20260810 15:16:18.734364 9085 catalog_manager.cc:5495] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1). New cstate: current_term: 1 leader_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } health_report { overall_health: UNKNOWN } } }
W20260810 15:16:18.781410 11373 tablet.cc:2369] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:18.785674 11425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.783217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.29.230.1:44389)
0810 15:16:18.783306 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.783308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.783317 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:18.783358 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.783359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.783378 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.783440 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:18.783606 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.784090 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.784854 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.785068 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.785133 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.785355 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:18.785476 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:18.785551 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.785594 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.785615 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:16:18.787556 11285 raft_consensus.cc:1261] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Refusing update from remote peer 8359f4ef0b1d4e9083872a01426bc7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:18.787556 11061 raft_consensus.cc:1261] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Refusing update from remote peer 8359f4ef0b1d4e9083872a01426bc7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:18.787932 11439 consensus_queue.cc:1035] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:18.788097 11438 consensus_queue.cc:1035] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:18.792774 11451 mvcc.cc:205] Tried to move back new op lower bound from 7316991913110896640 to 7316991912891846656. Current Snapshot: MvccSnapshot[applied={T|T < 7316991913110896640 or (T in {7316991913110896640})}]
I20260810 15:16:18.794265 9085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:18.794355 9085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:18.795619 9085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a3b0d6e183f4d08b5ad1f64f18b9e42: Sending DeleteTablet for 3 replicas of tablet a77d1f9607454488a334af40b8eb5810
I20260810 15:16:18.795959 11250 tablet_service.cc:1514] Processing DeleteTablet for tablet a77d1f9607454488a334af40b8eb5810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:18 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:47654
I20260810 15:16:18.796056 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet a77d1f9607454488a334af40b8eb5810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:18 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:50568
I20260810 15:16:18.796445 10777 tablet_service.cc:1514] Processing DeleteTablet for tablet a77d1f9607454488a334af40b8eb5810 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:18 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:58684
I20260810 15:16:18.796612 11462 tablet_replica.cc:330] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: stopping tablet replica
I20260810 15:16:18.796623 11461 tablet_replica.cc:330] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: stopping tablet replica
I20260810 15:16:18.796720 11462 raft_consensus.cc:2229] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:18.796819 11462 raft_consensus.cc:2258] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:18.796838 11461 raft_consensus.cc:2229] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:18.796996 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 9030
I20260810 15:16:18.797281 11462 ts_tablet_manager.cc:1905] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:18.799170 11462 ts_tablet_manager.cc:1918] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:18.799248 11462 log.cc:1196] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/a77d1f9607454488a334af40b8eb5810
I20260810 15:16:18.799654 11462 ts_tablet_manager.cc:1939] T a77d1f9607454488a334af40b8eb5810 P 2317b4eb6d794b208ca390c7924312cb: Deleting consensus metadata
I20260810 15:16:18.812340 11463 tablet_replica.cc:330] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: stopping tablet replica
I20260810 15:16:18.812359 11461 raft_consensus.cc:2258] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:18.812578 11463 raft_consensus.cc:2229] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:18.812741 11463 raft_consensus.cc:2258] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:18.814491 11463 ts_tablet_manager.cc:1905] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:18.814491 11461 ts_tablet_manager.cc:1905] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:18.816656 11463 ts_tablet_manager.cc:1918] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:18.816668 11461 ts_tablet_manager.cc:1918] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:18.816751 11463 log.cc:1196] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/a77d1f9607454488a334af40b8eb5810
I20260810 15:16:18.816861 11461 log.cc:1196] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/a77d1f9607454488a334af40b8eb5810
I20260810 15:16:18.817039 11463 ts_tablet_manager.cc:1939] T a77d1f9607454488a334af40b8eb5810 P 8359f4ef0b1d4e9083872a01426bc7cf: Deleting consensus metadata
I20260810 15:16:18.817250 11461 ts_tablet_manager.cc:1939] T a77d1f9607454488a334af40b8eb5810 P f8022693fa844a238473161008ed0b49: Deleting consensus metadata
W20260810 15:16:18.820374 11179 connection.cc:535] server connection from 127.0.0.1:47654 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 15:16:18.820395 10686 connection.cc:535] server connection from 127.0.0.1:58684 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 15:16:18.820559 10943 connection.cc:535] server connection from 127.0.0.1:50568 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:16:18.830150 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.822349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.29.230.61:35991)
0810 15:16:18.822402 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.822404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.822549 (+ 145us) server_negotiation.cc:407] Connection header received
0810 15:16:18.822674 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.822675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.822691 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.822720 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.823519 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.824324 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.825216 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.825518 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.825594 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.827611 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.827613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.827613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.827622 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.828990 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.829312 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.829313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.829313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.829370 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.829596 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.829597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.829598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.829748 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.829850 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.829958 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.830049 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:16:18.830794 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:18.822526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60694 (local address 127.29.230.60:36869)
0810 15:16:18.822570 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:18.822572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:18.822608 (+ 36us) server_negotiation.cc:407] Connection header received
0810 15:16:18.822674 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:18.822675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:18.822690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:18.822760 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:18.824446 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.825208 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.826160 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:18.826484 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:18.826555 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:18.827723 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:18.827725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:18.827725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:18.827733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:18.829238 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.829722 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.829723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.829724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.829776 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:18.830023 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:18.830025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:18.830025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:18.830230 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:18.830325 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:18.830621 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 15:16:18.830706 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:16:19.792529 11476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:19.792231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:34393)
0810 15:16:19.792361 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:19.792454 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
W20260810 15:16:19.792929 11370 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:37971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:19.793226 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:19.792967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:56973)
0810 15:16:19.793109 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:19.793182 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":52,"threads_started":1}
W20260810 15:16:19.793437 11141 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:37971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:19.793694 11478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:19.793457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:59087)
0810 15:16:19.793592 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:19.793659 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260810 15:16:19.793864 10912 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:37971 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.296018 11480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a3b0d6e183f4d08b5ad1f64f18b9e42)
I20260810 15:16:20.296162 11480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } }
I20260810 15:16:20.296439 11481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a3b0d6e183f4d08b5ad1f64f18b9e42)
I20260810 15:16:20.296451 11480 leader_election.cc:288] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971), 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61:35991)
I20260810 15:16:20.296839 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.296611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50270)
0810 15:16:20.296736 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.296794 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":51,"threads_started":1}
I20260810 15:16:20.296780 11481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } }
W20260810 15:16:20.297061 10525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.297339 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.297147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50276)
0810 15:16:20.297251 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.297303 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260810 15:16:20.297343 11481 leader_election.cc:288] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971), 722af23f24d4486288483b3105ef3dc1 (127.29.230.60:36869)
W20260810 15:16:20.297578 10201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.297648 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.297564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50286)
0810 15:16:20.297599 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.297622 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:16:20.297772 10525 leader_election.cc:334] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971): Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.298024 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.297939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50294)
0810 15:16:20.297975 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.298000 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:16:20.298141 10201 leader_election.cc:334] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971): Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.300037 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.296687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.29.230.61:35991)
0810 15:16:20.296778 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.296780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.296814 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:16:20.296900 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.296901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.296913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.296985 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:20.297191 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.297949 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.299355 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.299759 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.299810 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.299820 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.299964 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.299990 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":73,"thread_start_us":43,"threads_started":1}
I20260810 15:16:20.300073 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.296621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35991 (local address 127.0.0.1:32800)
0810 15:16:20.296740 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.296752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:20.296832 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:20.297048 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:20.297050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:20.297052 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:20.297112 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.297114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.298055 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.298056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.299253 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.299256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.299829 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.299831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.299900 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.299905 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:16:20.299960 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:16:20.300007 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":46,"threads_started":1}
I20260810 15:16:20.300447 10242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "722af23f24d4486288483b3105ef3dc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1966bcfcdb48443eaf4676295f979e71" is_pre_election: true
I20260810 15:16:20.300544 10242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 722af23f24d4486288483b3105ef3dc1 in term 1.
I20260810 15:16:20.300695 10528 leader_election.cc:302] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1966bcfcdb48443eaf4676295f979e71, 722af23f24d4486288483b3105ef3dc1; no voters: 2a3b0d6e183f4d08b5ad1f64f18b9e42
I20260810 15:16:20.300827 11480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 15:16:20.300876 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.297158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36869 (local address 127.0.0.1:45260)
0810 15:16:20.297853 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.297863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:20.297920 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:20.298066 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:20.298066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:20.298068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:20.298106 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.298107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.299017 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.299018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.300091 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.300094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.300677 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.300678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.300725 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.300728 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:20.300772 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:20.300812 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":20,"thread_start_us":42,"threads_started":1}
I20260810 15:16:20.300887 11480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2a3b0d6e183f4d08b5ad1f64f18b9e42)
I20260810 15:16:20.300921 11480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:16:20.300963 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.297347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.29.230.60:36869)
0810 15:16:20.297633 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.297635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.297913 (+ 278us) server_negotiation.cc:407] Connection header received
0810 15:16:20.297967 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.297968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.297978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.298045 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:20.298198 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.298914 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.300163 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.300580 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.300640 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.300649 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.300786 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.300823 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":82,"threads_started":1}
I20260810 15:16:20.301288 10572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1966bcfcdb48443eaf4676295f979e71" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "722af23f24d4486288483b3105ef3dc1" is_pre_election: true
I20260810 15:16:20.301743 11480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } }
I20260810 15:16:20.301869 10572 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1966bcfcdb48443eaf4676295f979e71 in current term 2: Already voted for candidate 722af23f24d4486288483b3105ef3dc1 in this term.
I20260810 15:16:20.301905 11480 leader_election.cc:288] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 election: Requested vote from peers 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971), 1966bcfcdb48443eaf4676295f979e71 (127.29.230.61:35991)
I20260810 15:16:20.302057 10242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "722af23f24d4486288483b3105ef3dc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1966bcfcdb48443eaf4676295f979e71"
I20260810 15:16:20.302102 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.301973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50302)
0810 15:16:20.302056 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.302082 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 15:16:20.302112 10205 leader_election.cc:302] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1966bcfcdb48443eaf4676295f979e71; no voters: 2a3b0d6e183f4d08b5ad1f64f18b9e42, 722af23f24d4486288483b3105ef3dc1
I20260810 15:16:20.302134 10242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:16:20.302373 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.302271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50316)
0810 15:16:20.302319 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.302350 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 15:16:20.302443 10525 leader_election.cc:334] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971): Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
I20260810 15:16:20.302830 10242 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 722af23f24d4486288483b3105ef3dc1 in term 2.
I20260810 15:16:20.302887 11481 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:16:20.303000 10528 leader_election.cc:302] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1966bcfcdb48443eaf4676295f979e71, 722af23f24d4486288483b3105ef3dc1; no voters: 2a3b0d6e183f4d08b5ad1f64f18b9e42
I20260810 15:16:20.303085 11480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:16:20.303220 11480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [term 2 LEADER]: Becoming Leader. State: Replica: 722af23f24d4486288483b3105ef3dc1, State: Running, Role: LEADER
I20260810 15:16:20.303305 11480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } }
I20260810 15:16:20.303632 11493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722af23f24d4486288483b3105ef3dc1. Latest consensus state: current_term: 2 leader_uuid: "722af23f24d4486288483b3105ef3dc1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:20.303706 11493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:20.303857 11495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:20.304338 11495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:20.304410 11495 catalog_manager.cc:1259] Loaded cluster ID: eba84c09f8ad41b6976183e903c3381c
I20260810 15:16:20.304440 11495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:20.304610 11495 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:20.304682 11495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Loaded TSK: 0
I20260810 15:16:20.304827 11495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:20.312009 10554 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:20.313735 10554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:20.314692 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.314583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50344)
0810 15:16:20.314633 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.314660 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:16:20.314708 10242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [term 2 FOLLOWER]: Refusing update from remote peer 722af23f24d4486288483b3105ef3dc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 15:16:20.315055 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.314929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.0.0.1:50358)
0810 15:16:20.314974 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.315003 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:16:20.315011 11493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 15:16:20.315142 10525 consensus_peers.cc:487] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 -> Peer 2a3b0d6e183f4d08b5ad1f64f18b9e42 (127.29.230.62:37971): Couldn't send request to peer 2a3b0d6e183f4d08b5ad1f64f18b9e42. Status: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 15:16:20.315642 11481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722af23f24d4486288483b3105ef3dc1. Latest consensus state: current_term: 2 leader_uuid: "722af23f24d4486288483b3105ef3dc1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:20.315747 11481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:16:20.316192 11480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "722af23f24d4486288483b3105ef3dc1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:20.316272 11480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:20.316428 11481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "722af23f24d4486288483b3105ef3dc1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "2a3b0d6e183f4d08b5ad1f64f18b9e42" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37971 } } peers { permanent_uuid: "1966bcfcdb48443eaf4676295f979e71" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35991 } attrs { promote: false } } peers { permanent_uuid: "722af23f24d4486288483b3105ef3dc1" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36869 } attrs { promote: false } } }
I20260810 15:16:20.316519 11481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1966bcfcdb48443eaf4676295f979e71 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:16:20.321128 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.318223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.29.230.1:44389)
0810 15:16:20.318328 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.318330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.318337 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:20.318362 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.318362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.318372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.318560 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:20.318608 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319382 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320308 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.320820 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320881 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.320894 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.321023 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.321060 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260810 15:16:20.321130 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.317983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44389 (local address 127.0.0.1:54202)
0810 15:16:20.318222 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.318233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:20.318279 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:20.318457 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:20.318458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:20.318459 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:20.318503 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.318505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319477 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.319478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.320211 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.320214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.320888 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.320970 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.320973 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:20.321018 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:20.321059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222}
I20260810 15:16:20.321309 11509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.318049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41382 (local address 127.29.230.3:33353)
0810 15:16:20.318249 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.318250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.318258 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:20.318287 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.318288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.318299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.318507 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:20.318543 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319012 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320302 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.320900 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.321174 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.321189 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.321206 (+ 17us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.321243 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
I20260810 15:16:20.321554 10777 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 (DEFAULT_TABLE table=test-table [id=05b03736ce924120a9f591ed3bb2869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:20.321622 11511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.318015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:46197 (local address 127.0.0.1:35408)
0810 15:16:20.318353 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.318360 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:20.318403 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:20.318587 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:20.318587 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:20.318589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:20.318618 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.318619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319518 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.319519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.320580 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.320583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.321390 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.321391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.321454 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.321458 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:20.321509 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:16:20.321562 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":131,"threads_started":1}
I20260810 15:16:20.321717 10777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8ff48e4643430eb2555477036ec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:20.321779 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.317921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:33353 (local address 127.0.0.1:41382)
0810 15:16:20.317962 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.317972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:20.318018 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:20.318385 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:20.318386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:20.318388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:20.318444 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.318445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319088 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.319089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.320204 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:20.320207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.320987 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:20.321047 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.321052 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:16:20.321080 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 15:16:20.321119 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 15:16:20.321802 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.318080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.29.230.2:46197)
0810 15:16:20.318208 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.318210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.318453 (+ 243us) server_negotiation.cc:407] Connection header received
0810 15:16:20.318482 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.318483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.318492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.318545 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:20.318687 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.319438 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.320674 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.321282 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.321343 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.321355 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.321519 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.321566 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260810 15:16:20.322069 11250 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 (DEFAULT_TABLE table=test-table [id=05b03736ce924120a9f591ed3bb2869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:20.322188 11250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8ff48e4643430eb2555477036ec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:20.323065 11516 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf: Bootstrap starting.
I20260810 15:16:20.323520 11516 tablet_bootstrap.cc:654] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:20.323635 11023 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 (DEFAULT_TABLE table=test-table [id=05b03736ce924120a9f591ed3bb2869d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:20.323761 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8ff48e4643430eb2555477036ec9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:20.324153 11516 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf: No bootstrap required, opened a new log
I20260810 15:16:20.324200 11516 ts_tablet_manager.cc:1397] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:20.324303 11516 raft_consensus.cc:348] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.324360 11516 raft_consensus.cc:374] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:20.324378 11516 raft_consensus.cc:729] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8359f4ef0b1d4e9083872a01426bc7cf, State: Initialized, Role: FOLLOWER
I20260810 15:16:20.324421 11516 consensus_queue.cc:260] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.324564 11516 ts_tablet_manager.cc:1428] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:20.324968 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.324743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:36887)
0810 15:16:20.324901 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.324936 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260810 15:16:20.325204 10910 heartbeater.cc:503] Master 127.29.230.60:36869 was elected leader, sending a full tablet report...
I20260810 15:16:20.325424 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.325220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:56575)
0810 15:16:20.325371 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.325397 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260810 15:16:20.325481 11521 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Bootstrap starting.
I20260810 15:16:20.325716 11529 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Bootstrap starting.
I20260810 15:16:20.325872 11521 tablet_bootstrap.cc:654] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:20.326045 11529 tablet_bootstrap.cc:654] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:20.326436 11521 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: No bootstrap required, opened a new log
I20260810 15:16:20.326480 11521 ts_tablet_manager.cc:1397] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:20.326603 11521 raft_consensus.cc:348] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.326659 11521 raft_consensus.cc:374] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:20.326676 11521 raft_consensus.cc:729] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8022693fa844a238473161008ed0b49, State: Initialized, Role: FOLLOWER
I20260810 15:16:20.326726 11529 tablet_bootstrap.cc:492] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: No bootstrap required, opened a new log
I20260810 15:16:20.326715 11521 consensus_queue.cc:260] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.326781 11529 ts_tablet_manager.cc:1397] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:20.326855 11521 ts_tablet_manager.cc:1428] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:20.326905 11529 raft_consensus.cc:348] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.327005 11529 raft_consensus.cc:374] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:20.327028 11529 raft_consensus.cc:729] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2317b4eb6d794b208ca390c7924312cb, State: Initialized, Role: FOLLOWER
I20260810 15:16:20.327095 11529 consensus_queue.cc:260] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.327239 11529 ts_tablet_manager.cc:1428] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:20.327280 11367 heartbeater.cc:503] Master 127.29.230.60:36869 was elected leader, sending a full tablet report...
I20260810 15:16:20.327597 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.327030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:39391)
0810 15:16:20.327527 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.327564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":76,"threads_started":1}
I20260810 15:16:20.327710 11139 heartbeater.cc:503] Master 127.29.230.60:36869 was elected leader, sending a full tablet report...
I20260810 15:16:20.328126 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.327918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:54725)
0810 15:16:20.328038 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.328101 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260810 15:16:20.328159 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.327588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:52205)
0810 15:16:20.328084 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.328124 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":78,"threads_started":1}
I20260810 15:16:20.328437 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.328353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:40465)
0810 15:16:20.328385 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.328415 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 15:16:20.504246 11523 raft_consensus.cc:482] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:20.504351 11523 raft_consensus.cc:504] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.504642 11523 leader_election.cc:288] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8022693fa844a238473161008ed0b49 (127.29.230.3:33353), 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
I20260810 15:16:20.504866 11285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8ff48e4643430eb2555477036ec9e8" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8022693fa844a238473161008ed0b49" is_pre_election: true
I20260810 15:16:20.504866 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8ff48e4643430eb2555477036ec9e8" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2317b4eb6d794b208ca390c7924312cb" is_pre_election: true
I20260810 15:16:20.504963 11061 raft_consensus.cc:2454] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 0.
I20260810 15:16:20.504964 11285 raft_consensus.cc:2454] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 0.
I20260810 15:16:20.505133 10688 leader_election.cc:302] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2317b4eb6d794b208ca390c7924312cb, 8359f4ef0b1d4e9083872a01426bc7cf; no voters:
I20260810 15:16:20.505234 11523 raft_consensus.cc:2781] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:20.505268 11523 raft_consensus.cc:482] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:20.505286 11523 raft_consensus.cc:3037] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:20.506022 11523 raft_consensus.cc:504] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.506137 11523 leader_election.cc:288] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 election: Requested vote from peers f8022693fa844a238473161008ed0b49 (127.29.230.3:33353), 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197)
I20260810 15:16:20.506317 11061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8ff48e4643430eb2555477036ec9e8" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2317b4eb6d794b208ca390c7924312cb"
I20260810 15:16:20.506331 11285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8ff48e4643430eb2555477036ec9e8" candidate_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8022693fa844a238473161008ed0b49"
I20260810 15:16:20.506397 11061 raft_consensus.cc:3037] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:20.506398 11285 raft_consensus.cc:3037] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:20.506965 11285 raft_consensus.cc:2454] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 1.
I20260810 15:16:20.507119 11061 raft_consensus.cc:2454] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8359f4ef0b1d4e9083872a01426bc7cf in term 1.
I20260810 15:16:20.507138 10686 leader_election.cc:302] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8359f4ef0b1d4e9083872a01426bc7cf, f8022693fa844a238473161008ed0b49; no voters:
I20260810 15:16:20.507216 11523 raft_consensus.cc:2781] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:20.507400 11523 raft_consensus.cc:686] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [term 1 LEADER]: Becoming Leader. State: Replica: 8359f4ef0b1d4e9083872a01426bc7cf, State: Running, Role: LEADER
I20260810 15:16:20.507453 11523 consensus_queue.cc:237] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } }
I20260810 15:16:20.508025 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.507889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:56679)
0810 15:16:20.507954 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.507991 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:16:20.508257 10554 catalog_manager.cc:5495] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1). New cstate: current_term: 1 leader_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8359f4ef0b1d4e9083872a01426bc7cf" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 } health_report { overall_health: UNKNOWN } } }
I20260810 15:16:20.547647 11514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.545487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.29.230.1:44389)
0810 15:16:20.545549 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:20.545550 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:20.545558 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:20.545580 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:20.545581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:20.545590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:20.545634 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:20.545757 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.546255 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.547025 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:20.547226 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:20.547318 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:20.547445 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:20.547485 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:20.547533 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:20.547553 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:16:20.547574 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:16:20.548547 11061 raft_consensus.cc:1261] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Refusing update from remote peer 8359f4ef0b1d4e9083872a01426bc7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:20.548597 11285 raft_consensus.cc:1261] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Refusing update from remote peer 8359f4ef0b1d4e9083872a01426bc7cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:20.548780 11543 consensus_queue.cc:1035] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2317b4eb6d794b208ca390c7924312cb" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:20.548907 11523 consensus_queue.cc:1035] T 4c8ff48e4643430eb2555477036ec9e8 P 8359f4ef0b1d4e9083872a01426bc7cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8022693fa844a238473161008ed0b49" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:20.549573 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.549354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:36323)
0810 15:16:20.549504 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.549540 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 15:16:20.549985 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.549884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:42869)
0810 15:16:20.549928 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.549954 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:16:20.550901 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.550815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:39075)
0810 15:16:20.550852 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.550875 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:16:20.551057 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.550971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.2:55211)
0810 15:16:20.551006 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.551029 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:16:20.551735 11551 mvcc.cc:205] Tried to move back new op lower bound from 7316991920325095424 to 7316991920158941184. Current Snapshot: MvccSnapshot[applied={T|T < 7316991920325095424 or (T in {7316991920325095424})}]
I20260810 15:16:20.551860 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.551596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.3:49495)
0810 15:16:20.551808 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.551836 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260810 15:16:20.551921 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.551842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:37971 (local address 127.29.230.1:55709)
0810 15:16:20.551878 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.551899 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:37971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:16:20.552358 10554 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:20.552462 10554 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:20.553663 10554 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 722af23f24d4486288483b3105ef3dc1: Sending DeleteTablet for 3 replicas of tablet 4c8ff48e4643430eb2555477036ec9e8
I20260810 15:16:20.553938 10777 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:20 UTC) from {username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:54202
I20260810 15:16:20.553998 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:20 UTC) from {username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:35408
I20260810 15:16:20.554102 11250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c8ff48e4643430eb2555477036ec9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:20 UTC) from {username='oryx', principal='oryx/127.29.230.60@KRBTEST.COM'} at 127.0.0.1:41382
I20260810 15:16:20.554232 11565 tablet_replica.cc:330] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: stopping tablet replica
I20260810 15:16:20.554282 11566 tablet_replica.cc:330] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: stopping tablet replica
I20260810 15:16:20.554294 11565 raft_consensus.cc:2229] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:20.554340 11565 raft_consensus.cc:2258] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:20.554344 11566 raft_consensus.cc:2229] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:20.554385 11566 raft_consensus.cc:2258] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:20.554584 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 10661
I20260810 15:16:20.561609 11565 ts_tablet_manager.cc:1905] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:20.563313 11565 ts_tablet_manager.cc:1918] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:20.563370 11565 log.cc:1196] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/4c8ff48e4643430eb2555477036ec9e8
I20260810 15:16:20.563572 11565 ts_tablet_manager.cc:1939] T 4c8ff48e4643430eb2555477036ec9e8 P 2317b4eb6d794b208ca390c7924312cb: Deleting consensus metadata
I20260810 15:16:20.563946 10526 catalog_manager.cc:4841] TS 2317b4eb6d794b208ca390c7924312cb (127.29.230.2:46197): tablet 4c8ff48e4643430eb2555477036ec9e8 (table test-table [id=05b03736ce924120a9f591ed3bb2869d]) successfully deleted
I20260810 15:16:20.564034 11566 ts_tablet_manager.cc:1905] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:20.565733 11566 ts_tablet_manager.cc:1918] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:20.565811 11566 log.cc:1196] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/4c8ff48e4643430eb2555477036ec9e8
I20260810 15:16:20.566061 11566 ts_tablet_manager.cc:1939] T 4c8ff48e4643430eb2555477036ec9e8 P f8022693fa844a238473161008ed0b49: Deleting consensus metadata
I20260810 15:16:20.566401 10525 catalog_manager.cc:4841] TS f8022693fa844a238473161008ed0b49 (127.29.230.3:33353): tablet 4c8ff48e4643430eb2555477036ec9e8 (table test-table [id=05b03736ce924120a9f591ed3bb2869d]) successfully deleted
I20260810 15:16:20.572897 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.572792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44389 (local address 127.0.0.1:54208)
0810 15:16:20.572838 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.572864 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:16:20.572986 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 10920
W20260810 15:16:20.573086 10528 catalog_manager.cc:4584] TS 8359f4ef0b1d4e9083872a01426bc7cf (127.29.230.1:44389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c8ff48e4643430eb2555477036ec9e8: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44389: connect: Connection refused (error 111)
I20260810 15:16:20.593796 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 11148
I20260810 15:16:20.632731 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.632597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44389 (local address 127.0.0.1:54212)
0810 15:16:20.632654 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.632684 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 15:16:20.633002 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:20.632932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44389 (local address 127.0.0.1:54228)
0810 15:16:20.632971 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:20.632986 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:16:20.641023 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 9982
I20260810 15:16:20.650861 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 10329
2026-08-10T15:16:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5100 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:20 dist-test-slave-gjrz krb5kdc[11571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:20 dist-test-slave-gjrz krb5kdc[11571](info): set up 2 sockets
Aug 10 15:16:20 dist-test-slave-gjrz krb5kdc[11571](info): commencing operation
krb5kdc: starting...
W20260810 15:16:21.831528 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:21 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab.
Entry for principal oryx/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab.
I20260810 15:16:21.950291 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:45377
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab
--principal=oryx/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:45377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:22.032089 11601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:22.032310 11601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:22.032341 11601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:22.033488 11601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:22.033529 11601 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 15:16:22.033546 11601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:22.033564 11601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:22.033579 11601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:22.033596 11601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:22.035167 11601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:45377
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.62.keytab
--principal=oryx/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:22.035450 11601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:22.035712 11601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:22.040452 11611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.040478 11609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.040454 11608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:22.040798 11601 server_base.cc:1034] running on GCE node
I20260810 15:16:22.041049 11601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:22.041349 11601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:22.042507 11601 hybrid_clock.cc:648] HybridClock initialized: now 1786374982042493 us; error 31 us; skew 500 ppm
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:22.047858 11601 init.cc:377] Logged in from keytab as oryx/127.29.230.62@KRBTEST.COM (short username oryx)
I20260810 15:16:22.048343 11601 webserver.cc:460] Webserver started at http://127.29.230.62:36523/ using document root <none> and password file <none>
I20260810 15:16:22.048569 11601 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:22.048627 11601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:22.048763 11601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:22.049890 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "493745a1c58348ba81776d4a38260c74"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "abcd590957e2fb62053ee602a4a44c2b"
server_key_iv: "b0692a45c57fcb86725c3b50d85b299a"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.050240 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "493745a1c58348ba81776d4a38260c74"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "abcd590957e2fb62053ee602a4a44c2b"
server_key_iv: "b0692a45c57fcb86725c3b50d85b299a"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.051594 11601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:22.052311 11624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.052461 11601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:22.052549 11601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "493745a1c58348ba81776d4a38260c74"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "abcd590957e2fb62053ee602a4a44c2b"
server_key_iv: "b0692a45c57fcb86725c3b50d85b299a"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.052631 11601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:22.070472 11601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:22.072556 11601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:22.072700 11601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:22.077450 11601 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:45377
I20260810 15:16:22.077543 11735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:45377 every 8 connection(s)
I20260810 15:16:22.077829 11601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:22.078511 11738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.080928 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Bootstrap starting.
I20260810 15:16:22.081511 11738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.081761 11738 log.cc:824] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:22.082446 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: No bootstrap required, opened a new log
I20260810 15:16:22.084591 11738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } }
I20260810 15:16:22.084735 11738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.084756 11738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493745a1c58348ba81776d4a38260c74, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.084861 11738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } }
I20260810 15:16:22.084913 11738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:22.084933 11738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:22.084965 11738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.085441 11738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } }
I20260810 15:16:22.085529 11738 leader_election.cc:302] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 493745a1c58348ba81776d4a38260c74; no voters:
I20260810 15:16:22.085595 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 11601
I20260810 15:16:22.085695 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:22.085698 11738 leader_election.cc:288] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:22.085743 11745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:22.085901 30616 external_mini_cluster.cc:1428] Setting key 81e773237dc8d1482f14cc288e8e6601
I20260810 15:16:22.085920 11745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [term 1 LEADER]: Becoming Leader. State: Replica: 493745a1c58348ba81776d4a38260c74, State: Running, Role: LEADER
I20260810 15:16:22.085964 11738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:22.086004 11745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } }
I20260810 15:16:22.086567 11749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 493745a1c58348ba81776d4a38260c74. Latest consensus state: current_term: 1 leader_uuid: "493745a1c58348ba81776d4a38260c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } } }
I20260810 15:16:22.086659 11749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:22.087168 11760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:22.087764 11760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:22.088244 11746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "493745a1c58348ba81776d4a38260c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "493745a1c58348ba81776d4a38260c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 45377 } } }
I20260810 15:16:22.088320 11746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:22.089303 11760 catalog_manager.cc:1347] Generated new cluster ID: 75e3b9d54bdc4449840946890488168f
I20260810 15:16:22.089352 11760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.097687 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.086430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.29.230.62:45377)
0810 15:16:22.086623 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.086633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.086973 (+ 340us) server_negotiation.cc:407] Connection header received
0810 15:16:22.087198 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.087206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.087296 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.087421 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.088184 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.089424 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.090473 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.091097 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.091191 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.093966 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.093971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.093971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.093985 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.095947 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.096325 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.096328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.096329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.096443 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.096630 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.096631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.096632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.096971 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.097140 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.097210 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.097461 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260810 15:16:22.106578 11760 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:22.107314 11760 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:22.113040 11760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Generated new TSK 0
I20260810 15:16:22.113229 11760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
I20260810 15:16:22.140084 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:45377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:22.221000 11782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:22.221194 11782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:22.221215 11782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:22.222344 11782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:22.222384 11782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:22.222402 11782 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:16:22.222420 11782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:22.223955 11782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:22.224181 11782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:22.224392 11782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:22.229250 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.229336 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.229233 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:22.229390 11782 server_base.cc:1034] running on GCE node
I20260810 15:16:22.229609 11782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:22.229812 11782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:22.230945 11782 hybrid_clock.cc:648] HybridClock initialized: now 1786374982230937 us; error 35 us; skew 500 ppm
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:22.233425 11782 init.cc:377] Logged in from keytab as oryx/127.29.230.1@KRBTEST.COM (short username oryx)
I20260810 15:16:22.233835 11782 webserver.cc:460] Webserver started at http://127.29.230.1:33481/ using document root <none> and password file <none>
I20260810 15:16:22.234028 11782 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:22.234067 11782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:22.234156 11782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:22.235112 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "4027b619891646989611b2f614641433"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "b721ebf3403cd6e14adc3a5c951d92aa"
server_key_iv: "36d3f030867728279dbf0f4774098f40"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.235456 11782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "4027b619891646989611b2f614641433"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "b721ebf3403cd6e14adc3a5c951d92aa"
server_key_iv: "36d3f030867728279dbf0f4774098f40"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.236763 11782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:22.237497 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.237653 11782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:22.237741 11782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "4027b619891646989611b2f614641433"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "b721ebf3403cd6e14adc3a5c951d92aa"
server_key_iv: "36d3f030867728279dbf0f4774098f40"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.237805 11782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:22.254333 11782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:22.256554 11782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:22.256690 11782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:22.257212 11782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:22.257254 11782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.257282 11782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:22.257308 11782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.263963 11782 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:45819
I20260810 15:16:22.264034 11986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:45819 every 8 connection(s)
I20260810 15:16:22.264374 11782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:22.264894 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 11782
I20260810 15:16:22.265029 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:22.265241 30616 external_mini_cluster.cc:1428] Setting key 9d0bc1d96a16fccb60f61076bf37b880
I20260810 15:16:22.267112 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.257627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47541 (local address 127.29.230.62:45377)
0810 15:16:22.257681 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.257684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.257860 (+ 176us) server_negotiation.cc:407] Connection header received
0810 15:16:22.258331 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.258332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.258345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.258425 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.259235 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.260016 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.261049 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.261256 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.261314 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.264137 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.264139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.264139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.264150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.265682 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.266163 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.266164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.266165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.266218 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.266501 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.266502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.266503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.266643 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.266706 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.266943 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.267032 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:16:22.267210 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.257593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.1:47541)
0810 15:16:22.257741 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.257779 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.258243 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.258502 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.258505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.258800 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.259117 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.259124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.260135 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.260136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.260961 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.260963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.261337 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.261338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.261468 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.261886 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.261904 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.264023 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.265747 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.265754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.265755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.266084 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.266270 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.266271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.266271 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.266419 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.266701 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.266708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.266824 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.266917 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.267045 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260810 15:16:22.267732 11988 heartbeater.cc:347] Connected to a master server at 127.29.230.62:45377
I20260810 15:16:22.267818 11988 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:22.268008 11988 heartbeater.cc:511] Master 127.29.230.62:45377 requested a full tablet report, sending...
I20260810 15:16:22.268563 11651 ts_manager.cc:194] Registered new tserver with Master: 4027b619891646989611b2f614641433 (127.29.230.1:45819)
I20260810 15:16:22.269327 11651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:47541
WARNING: no policy specified for oryx/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:16:22.275863 12002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.269636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.1:46603)
0810 15:16:22.269762 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.269773 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.269845 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.269960 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.269961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.270150 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.270209 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.270211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.270877 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.270878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.271706 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.271709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.272103 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.272104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.272171 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.272557 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.272562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.272913 (+ 351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.274324 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.274325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.274326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.274533 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.274775 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.274776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.274776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.274842 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.275544 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.275545 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.275593 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.275662 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.275763 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260810 15:16:22.275930 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.269698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:46603 (local address 127.29.230.62:45377)
0810 15:16:22.269737 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.269739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.269836 (+ 97us) server_negotiation.cc:407] Connection header received
0810 15:16:22.269890 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.269891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.269901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.269957 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.270284 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.270773 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.271806 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.272025 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.272086 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.273020 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.273021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.273022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.273027 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.274242 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.274618 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.274619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.274619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.274698 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.275325 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.275326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.275327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.275476 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.275547 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.275722 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.275851 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
I20260810 15:16:22.292426 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:45377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:22.372258 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:22.372448 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:22.372481 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:22.373605 12008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:22.373644 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:22.373672 12008 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:16:22.373695 12008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:22.375193 12008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:22.375432 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:22.375634 12008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:22.380394 12015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.380403 12016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.380467 12018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:22.380681 12008 server_base.cc:1034] running on GCE node
I20260810 15:16:22.380878 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:22.381102 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:22.382243 12008 hybrid_clock.cc:648] HybridClock initialized: now 1786374982382231 us; error 28 us; skew 500 ppm
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:22.384598 12008 init.cc:377] Logged in from keytab as oryx/127.29.230.2@KRBTEST.COM (short username oryx)
I20260810 15:16:22.385004 12008 webserver.cc:460] Webserver started at http://127.29.230.2:43089/ using document root <none> and password file <none>
I20260810 15:16:22.385190 12008 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:22.385232 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:22.385350 12008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:22.386389 12008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "33b57e1149bc4caaaeeab72430d508cb"
server_key_iv: "3fe025f65c0c6b96c588af294d4c345e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.386745 12008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "33b57e1149bc4caaaeeab72430d508cb"
server_key_iv: "3fe025f65c0c6b96c588af294d4c345e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.388121 12008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:16:22.388870 12028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.389055 12008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:22.389139 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "33b57e1149bc4caaaeeab72430d508cb"
server_key_iv: "3fe025f65c0c6b96c588af294d4c345e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.389199 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:22.409051 12008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:22.410830 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:22.410962 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:22.411490 12008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:22.411531 12008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.411559 12008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:22.411576 12008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.418264 12008 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:36079
I20260810 15:16:22.418351 12221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:36079 every 8 connection(s)
I20260810 15:16:22.418612 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:22.420620 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.411889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:37841 (local address 127.29.230.62:45377)
0810 15:16:22.411956 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.411959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.412177 (+ 218us) server_negotiation.cc:407] Connection header received
0810 15:16:22.412798 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.412799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.412812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.412875 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.413732 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.414519 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.415560 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.415777 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.415834 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.418126 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.418128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.418128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.418136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.419593 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.419925 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.419925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.419926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.419969 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.420174 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.420175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.420175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.420273 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.420317 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.420503 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.420559 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 15:16:22.420775 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.411860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.2:37841)
0810 15:16:22.412031 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.412078 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.412698 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.412952 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.412957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.413330 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.413617 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.413624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.414599 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.414601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.415470 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.415473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.415864 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.415865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.415995 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.416434 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.416443 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.418013 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.419668 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.419671 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.419672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.419872 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.420043 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.420043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.420043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.420130 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.420309 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.420311 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.420412 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.420483 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.420597 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260810 15:16:22.421222 12223 heartbeater.cc:347] Connected to a master server at 127.29.230.62:45377
I20260810 15:16:22.421304 12223 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:22.421496 12223 heartbeater.cc:511] Master 127.29.230.62:45377 requested a full tablet report, sending...
I20260810 15:16:22.421917 11649 ts_manager.cc:194] Registered new tserver with Master: 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:22.422317 11649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:37841
I20260810 15:16:22.427016 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.422221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.2:33385)
0810 15:16:22.422343 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.422353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.422438 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.422550 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.422551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.422742 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.422809 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.422812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.423469 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.423470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.423944 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.423945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.424295 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.424296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.424343 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.424676 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.424679 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.424923 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.426252 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.426253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.426253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.426405 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.426552 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.426552 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.426553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.426600 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.426798 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.426798 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.426831 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.426870 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.426948 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260810 15:16:22.427021 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.422293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:33385 (local address 127.29.230.62:45377)
0810 15:16:22.422334 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.422335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.422405 (+ 70us) server_negotiation.cc:407] Connection header received
0810 15:16:22.422494 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.422495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.422505 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.422545 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.422893 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.423394 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.424032 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.424235 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.424289 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.424999 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.425000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.425000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.425005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.426183 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.426466 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.426467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.426467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.426510 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.426639 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.426640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.426640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.426757 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.426801 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.426893 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.426945 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:16:22.427436 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 12008
I20260810 15:16:22.427552 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:22.427767 30616 external_mini_cluster.cc:1428] Setting key 199f543b6396668084c09d0e1aff22e1
WARNING: no policy specified for oryx/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
I20260810 15:16:22.454552 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:45377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:22.535501 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:22.535703 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:22.535737 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:22.536846 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:22.536886 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:22.536914 12237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 15:16:22.536938 12237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:22.538414 12237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.230.62:45377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:22.538636 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:22.538836 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:22.544036 12251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:22.544027 12237 server_base.cc:1034] running on GCE node
W20260810 15:16:22.543962 12246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:22.543968 12248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:22.544474 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:22.544746 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:22.545914 12237 hybrid_clock.cc:648] HybridClock initialized: now 1786374982545899 us; error 30 us; skew 500 ppm
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:22.548461 12237 init.cc:377] Logged in from keytab as oryx/127.29.230.3@KRBTEST.COM (short username oryx)
I20260810 15:16:22.548889 12237 webserver.cc:460] Webserver started at http://127.29.230.3:46367/ using document root <none> and password file <none>
I20260810 15:16:22.549083 12237 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:22.549131 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:22.549256 12237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:22.550308 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "4f33600ce69b48a0a5b2d011199ebd0c"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "6c5349f44ec1644653587e502830be4d"
server_key_iv: "a2ead90ade5a981e3bd5bf9ea29ce28e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.550652 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "4f33600ce69b48a0a5b2d011199ebd0c"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "6c5349f44ec1644653587e502830be4d"
server_key_iv: "a2ead90ade5a981e3bd5bf9ea29ce28e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.551951 12237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 15:16:22.552661 12262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.552820 12237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:22.552904 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "4f33600ce69b48a0a5b2d011199ebd0c"
format_stamp: "Formatted at 2026-08-10 15:16:22 on dist-test-slave-gjrz"
server_key: "6c5349f44ec1644653587e502830be4d"
server_key_iv: "a2ead90ade5a981e3bd5bf9ea29ce28e"
server_key_version: "encryptionkey@0"
I20260810 15:16:22.552964 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:22.573524 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:22.575734 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:22.575873 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:22.576390 12237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:22.576431 12237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.576460 12237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:22.576486 12237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.582966 12237 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:39813
I20260810 15:16:22.583037 12458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:39813 every 8 connection(s)
I20260810 15:16:22.583387 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.586715 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.576853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:43087 (local address 127.29.230.62:45377)
0810 15:16:22.576918 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.576920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.577187 (+ 267us) server_negotiation.cc:407] Connection header received
0810 15:16:22.577733 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.577735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.577749 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.577803 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.578658 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.579479 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.580985 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.581273 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.581354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.583965 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.583967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.583967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.583975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.585351 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.585747 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.585749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.585749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.585808 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.586073 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.586074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.586075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.586199 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.586261 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.586530 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.586626 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:22.586901 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.576857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.3:43087)
0810 15:16:22.577048 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.577097 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.577594 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.577881 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.577886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.578252 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.578544 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.578550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.579579 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.579580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.580876 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.580880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.581379 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.581380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.581512 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.582074 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.582085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.583862 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.585428 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.585436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.585438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.585657 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.585863 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.585863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.585864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.586000 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.586256 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.586264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.586409 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.586499 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.586663 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260810 15:16:22.587363 12459 heartbeater.cc:347] Connected to a master server at 127.29.230.62:45377
I20260810 15:16:22.587615 12459 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:22.587819 12459 heartbeater.cc:511] Master 127.29.230.62:45377 requested a full tablet report, sending...
I20260810 15:16:22.588241 11651 ts_manager.cc:194] Registered new tserver with Master: 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813)
I20260810 15:16:22.588610 11651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:43087
I20260810 15:16:22.590013 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 12237
I20260810 15:16:22.590107 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:22.590301 30616 external_mini_cluster.cc:1428] Setting key 467963de64eb4e6c7972547a021a9467
I20260810 15:16:22.591125 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:16:22.594862 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.588825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.29.230.3:33095)
0810 15:16:22.589390 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.589400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.589453 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.589551 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.589552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.589702 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.589750 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.589751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.590423 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.590424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.591118 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.591121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.591529 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.591530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.591611 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.592020 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.592025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.592359 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.593931 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.593932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.593933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.594142 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.594313 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.594314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.594314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.594384 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.594619 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.594620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.594667 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.594719 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.594797 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":75,"threads_started":1}
I20260810 15:16:22.594904 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.588879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:33095 (local address 127.29.230.62:45377)
0810 15:16:22.588926 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.588928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.589460 (+ 532us) server_negotiation.cc:407] Connection header received
0810 15:16:22.589485 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.589485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.589495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.589548 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.589794 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.590343 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.591198 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.591460 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.591510 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.592452 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.592454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.592454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.592460 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.593853 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.594210 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.594211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.594211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.594259 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.594446 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.594447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.594447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.594558 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.594621 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.594752 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.594823 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:16:22.613034 11756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:22.607869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.29.230.62:45377)
0810 15:16:22.607928 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.607930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.607937 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:22.607968 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.607968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.607977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.608022 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.608549 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.609055 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.609759 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.609943 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.610003 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.612897 (+ 2894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38722: BlockingRecv error: recv got EOF from 127.0.0.1:38722 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.619905 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.613823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.29.230.62:45377)
0810 15:16:22.613865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.613867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.613929 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:16:22.614005 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.614006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.614015 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.614060 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.614483 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.614950 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.615645 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.615826 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.615878 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.617826 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.617828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.617828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.617836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.618902 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.619202 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.619203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.619204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.619255 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.619446 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.619447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.619448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.619556 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.619641 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.619741 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.619817 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:22.621351 11651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:22.623008 11651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:22.629431 12503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39813 (local address 127.0.0.1:60408)
0810 15:16:22.625741 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.625750 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.625800 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.626063 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.626067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.626069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.626133 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.626135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.627042 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.627042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.628251 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.628254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629141 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.629232 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629246 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.629310 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.629358 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":76,"threads_started":1}
I20260810 15:16:22.629577 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36079 (local address 127.0.0.1:42456)
0810 15:16:22.625438 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.625485 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.625561 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.626069 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.626070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.626072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.626150 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.626152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.627501 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.627502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.628528 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.628531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629361 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.629421 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629424 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.629481 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.629518 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260810 15:16:22.629590 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.29.230.3:39813)
0810 15:16:22.625826 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.625832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.625841 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:22.625884 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.625887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.625902 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.625975 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.626215 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.626966 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.628331 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629047 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629122 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629196 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.629341 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.629383 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260810 15:16:22.629822 12502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45819 (local address 127.0.0.1:59254)
0810 15:16:22.625866 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.625874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.625905 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.626275 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.626276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.626277 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.626309 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.626311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.627508 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.627509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.628446 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.628449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629610 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.629667 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629671 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.629715 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.629749 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":33,"thread_start_us":68,"threads_started":1}
I20260810 15:16:22.629868 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.29.230.2:36079)
0810 15:16:22.625877 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.625882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.625894 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:16:22.625935 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.625937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.625954 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.626230 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.626291 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.627401 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.628594 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629285 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629350 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629416 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.629496 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.629775 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":112,"threads_started":1}
I20260810 15:16:22.630055 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.625784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.29.230.1:45819)
0810 15:16:22.626109 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.626113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.626123 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:22.626158 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.626160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.626175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.626226 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.626360 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.627422 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.628532 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.629050 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.629124 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.629203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.629920 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.629964 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":75,"threads_started":1}
I20260810 15:16:22.630357 12351 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe (DEFAULT_TABLE table=test-table [id=e4d2be5acadd4713962c214409b948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.630487 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe (DEFAULT_TABLE table=test-table [id=e4d2be5acadd4713962c214409b948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.630695 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab143b2735f4ee2b5bec7a5fa120afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.630827 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab143b2735f4ee2b5bec7a5fa120afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.633875 12509 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Bootstrap starting.
I20260810 15:16:22.634359 12510 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Bootstrap starting.
I20260810 15:16:22.634596 12509 tablet_bootstrap.cc:654] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.634652 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe (DEFAULT_TABLE table=test-table [id=e4d2be5acadd4713962c214409b948e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.634904 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab143b2735f4ee2b5bec7a5fa120afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.634910 12509 log.cc:824] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:22.634953 12510 tablet_bootstrap.cc:654] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.635247 12510 log.cc:824] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:22.635756 12509 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: No bootstrap required, opened a new log
I20260810 15:16:22.635833 12509 ts_tablet_manager.cc:1397] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:22.636130 12510 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: No bootstrap required, opened a new log
I20260810 15:16:22.636185 12510 ts_tablet_manager.cc:1397] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:22.637794 12509 raft_consensus.cc:348] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.637953 12509 raft_consensus.cc:374] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.638005 12509 raft_consensus.cc:729] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.638146 12509 consensus_queue.cc:260] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.638219 12510 raft_consensus.cc:348] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.638383 12510 raft_consensus.cc:374] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.638427 12510 raft_consensus.cc:729] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.638442 12509 ts_tablet_manager.cc:1428] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:16:22.638464 12459 heartbeater.cc:503] Master 127.29.230.62:45377 was elected leader, sending a full tablet report...
I20260810 15:16:22.638535 12510 consensus_queue.cc:260] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.638638 12520 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Bootstrap starting.
I20260810 15:16:22.638801 12510 ts_tablet_manager.cc:1428] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:16:22.638924 12223 heartbeater.cc:503] Master 127.29.230.62:45377 was elected leader, sending a full tablet report...
I20260810 15:16:22.639249 12520 tablet_bootstrap.cc:654] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.639619 12520 log.cc:824] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:22.640348 12520 tablet_bootstrap.cc:492] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: No bootstrap required, opened a new log
I20260810 15:16:22.640410 12520 ts_tablet_manager.cc:1397] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:16:22.641768 12520 raft_consensus.cc:348] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.641906 12520 raft_consensus.cc:374] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.641952 12520 raft_consensus.cc:729] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027b619891646989611b2f614641433, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.642083 12520 consensus_queue.cc:260] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.642329 12520 ts_tablet_manager.cc:1428] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:16:22.642452 11988 heartbeater.cc:503] Master 127.29.230.62:45377 was elected leader, sending a full tablet report...
W20260810 15:16:22.669134 12224 tablet.cc:2369] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:22.696628 12526 raft_consensus.cc:482] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:22.696758 12526 raft_consensus.cc:504] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.697077 12526 leader_election.cc:288] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079), 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813)
I20260810 15:16:22.700364 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.697265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36079 (local address 127.29.230.1:47519)
0810 15:16:22.697314 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.697333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.697380 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.697525 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.697526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.697528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.697596 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.697597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.698452 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.698453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.699404 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.699407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.700101 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.700102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.700195 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.700200 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.700252 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.700294 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 15:16:22.700371 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.697328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47519 (local address 127.29.230.2:36079)
0810 15:16:22.697379 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.697381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.697394 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:16:22.697431 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.697432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.697441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.697517 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.697655 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.698376 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.699509 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.700001 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.700062 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.700070 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.700263 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.700302 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:16:22.700508 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.697459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:36201 (local address 127.29.230.3:39813)
0810 15:16:22.697544 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.697546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.697714 (+ 168us) server_negotiation.cc:407] Connection header received
0810 15:16:22.697742 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.697743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.697753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.697803 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.697927 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.698573 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.699758 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.700153 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.700230 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.700239 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.700396 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.700439 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 15:16:22.700663 12528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.697378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39813 (local address 127.29.230.1:36201)
0810 15:16:22.697641 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.697650 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.697707 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.697811 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.697811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.697813 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.697877 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.697877 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.698635 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.698636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.699651 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.699655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.700240 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.700241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.700317 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.700321 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.700377 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.700538 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":212,"threads_started":1}
I20260810 15:16:22.700672 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab143b2735f4ee2b5bec7a5fa120afe" candidate_uuid: "4027b619891646989611b2f614641433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" is_pre_election: true
I20260810 15:16:22.700815 12148 raft_consensus.cc:2454] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4027b619891646989611b2f614641433 in term 0.
I20260810 15:16:22.700984 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab143b2735f4ee2b5bec7a5fa120afe" candidate_uuid: "4027b619891646989611b2f614641433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" is_pre_election: true
I20260810 15:16:22.701058 11806 leader_election.cc:302] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4027b619891646989611b2f614641433; no voters:
I20260810 15:16:22.701102 12383 raft_consensus.cc:2454] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4027b619891646989611b2f614641433 in term 0.
I20260810 15:16:22.701169 12526 raft_consensus.cc:2781] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:22.701226 12526 raft_consensus.cc:482] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:22.701246 12526 raft_consensus.cc:3037] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.702175 12526 raft_consensus.cc:504] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.702288 12526 leader_election.cc:288] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [CANDIDATE]: Term 1 election: Requested vote from peers 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079), 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813)
I20260810 15:16:22.702461 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab143b2735f4ee2b5bec7a5fa120afe" candidate_uuid: "4027b619891646989611b2f614641433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
I20260810 15:16:22.702486 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab143b2735f4ee2b5bec7a5fa120afe" candidate_uuid: "4027b619891646989611b2f614641433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f33600ce69b48a0a5b2d011199ebd0c"
I20260810 15:16:22.702528 12148 raft_consensus.cc:3037] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.702574 12383 raft_consensus.cc:3037] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.703219 12148 raft_consensus.cc:2454] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4027b619891646989611b2f614641433 in term 1.
I20260810 15:16:22.703295 12383 raft_consensus.cc:2454] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4027b619891646989611b2f614641433 in term 1.
I20260810 15:16:22.703385 11806 leader_election.cc:302] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4027b619891646989611b2f614641433; no voters:
I20260810 15:16:22.703492 12526 raft_consensus.cc:2781] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:22.703637 12526 raft_consensus.cc:686] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 1 LEADER]: Becoming Leader. State: Replica: 4027b619891646989611b2f614641433, State: Running, Role: LEADER
I20260810 15:16:22.703740 12526 consensus_queue.cc:237] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } }
I20260810 15:16:22.704559 11651 catalog_manager.cc:5495] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4027b619891646989611b2f614641433 (127.29.230.1). New cstate: current_term: 1 leader_uuid: "4027b619891646989611b2f614641433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } health_report { overall_health: HEALTHY } } }
I20260810 15:16:22.721848 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.719607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.29.230.1:45819)
0810 15:16:22.719669 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.719671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.719680 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:22.719717 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.719718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.719734 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.719791 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:22.719911 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.720391 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.721130 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.721314 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.721368 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.721545 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:22.721646 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:22.721715 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.721752 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.721773 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:22.723670 12148 raft_consensus.cc:1261] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Refusing update from remote peer 4027b619891646989611b2f614641433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:22.723716 12383 raft_consensus.cc:1261] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Refusing update from remote peer 4027b619891646989611b2f614641433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:22.723982 12535 consensus_queue.cc:1035] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:22.724110 12526 consensus_queue.cc:1035] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:22.726606 12540 mvcc.cc:205] Tried to move back new op lower bound from 7316991929233530880 to 7316991929155129344. Current Snapshot: MvccSnapshot[applied={T|T < 7316991929233530880}]
I20260810 15:16:22.727047 12544 mvcc.cc:205] Tried to move back new op lower bound from 7316991929233530880 to 7316991929155129344. Current Snapshot: MvccSnapshot[applied={T|T < 7316991929233530880}]
I20260810 15:16:22.727597 12541 mvcc.cc:205] Tried to move back new op lower bound from 7316991929233530880 to 7316991929155129344. Current Snapshot: MvccSnapshot[applied={T|T < 7316991929233530880 or (T in {7316991929233530880})}]
I20260810 15:16:22.729015 11651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:22.729102 11651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:22.729789 11651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Sending DeleteTablet for 3 replicas of tablet 9ab143b2735f4ee2b5bec7a5fa120afe
I20260810 15:16:22.730121 12351 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:22 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260810 15:16:22.730170 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:22 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:59254
I20260810 15:16:22.730213 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab143b2735f4ee2b5bec7a5fa120afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:22 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260810 15:16:22.730420 12559 tablet_replica.cc:330] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: stopping tablet replica
I20260810 15:16:22.730458 12555 tablet_replica.cc:330] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: stopping tablet replica
I20260810 15:16:22.730522 12559 raft_consensus.cc:2229] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:22.730527 12555 raft_consensus.cc:2229] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:22.730630 12559 raft_consensus.cc:2258] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:22.730648 11651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:22.730789 11651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:22.730799 12556 tablet_replica.cc:330] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: stopping tablet replica
I20260810 15:16:22.730856 12555 raft_consensus.cc:2258] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:22.730880 12556 raft_consensus.cc:2229] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:22.731016 12556 raft_consensus.cc:2258] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:22.731318 12559 ts_tablet_manager.cc:1905] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:22.731487 12556 ts_tablet_manager.cc:1905] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:22.732829 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 544feca7ece64998bd9e3d93d3387a00 (DEFAULT_TABLE table=test-table [id=42c5523a88694755b40aa5bcdce9e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.732951 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544feca7ece64998bd9e3d93d3387a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.733130 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 544feca7ece64998bd9e3d93d3387a00 (DEFAULT_TABLE table=test-table [id=42c5523a88694755b40aa5bcdce9e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.733228 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544feca7ece64998bd9e3d93d3387a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.733330 12559 ts_tablet_manager.cc:1918] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:22.733395 12559 log.cc:1196] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/9ab143b2735f4ee2b5bec7a5fa120afe
I20260810 15:16:22.733639 12559 ts_tablet_manager.cc:1939] T 9ab143b2735f4ee2b5bec7a5fa120afe P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting consensus metadata
I20260810 15:16:22.734043 11625 catalog_manager.cc:4841] TS 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079): tablet 9ab143b2735f4ee2b5bec7a5fa120afe (table test-table [id=e4d2be5acadd4713962c214409b948e0]) successfully deleted
I20260810 15:16:22.734223 12520 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Bootstrap starting.
I20260810 15:16:22.734367 12556 ts_tablet_manager.cc:1918] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:22.734449 12556 log.cc:1196] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/9ab143b2735f4ee2b5bec7a5fa120afe
I20260810 15:16:22.734592 12351 tablet_service.cc:1467] Processing CreateTablet for tablet 544feca7ece64998bd9e3d93d3387a00 (DEFAULT_TABLE table=test-table [id=42c5523a88694755b40aa5bcdce9e19e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:22.734649 12520 tablet_bootstrap.cc:654] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.734705 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544feca7ece64998bd9e3d93d3387a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.734758 12556 ts_tablet_manager.cc:1939] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4027b619891646989611b2f614641433: Deleting consensus metadata
I20260810 15:16:22.734964 12510 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Bootstrap starting.
I20260810 15:16:22.735127 11625 catalog_manager.cc:4841] TS 4027b619891646989611b2f614641433 (127.29.230.1:45819): tablet 9ab143b2735f4ee2b5bec7a5fa120afe (table test-table [id=e4d2be5acadd4713962c214409b948e0]) successfully deleted
I20260810 15:16:22.735359 12510 tablet_bootstrap.cc:654] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.735620 12520 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: No bootstrap required, opened a new log
I20260810 15:16:22.735668 12520 ts_tablet_manager.cc:1397] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:22.735782 12520 raft_consensus.cc:348] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.735828 12520 raft_consensus.cc:374] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.735841 12520 raft_consensus.cc:729] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027b619891646989611b2f614641433, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.735875 12520 consensus_queue.cc:260] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.735952 12520 ts_tablet_manager.cc:1428] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.736030 12509 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Bootstrap starting.
I20260810 15:16:22.736444 12509 tablet_bootstrap.cc:654] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.736982 12555 ts_tablet_manager.cc:1905] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:22.737039 12510 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: No bootstrap required, opened a new log
I20260810 15:16:22.737087 12510 ts_tablet_manager.cc:1397] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:16:22.737175 12509 tablet_bootstrap.cc:492] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: No bootstrap required, opened a new log
I20260810 15:16:22.737222 12509 ts_tablet_manager.cc:1397] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:22.737233 12510 raft_consensus.cc:348] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.737308 12510 raft_consensus.cc:374] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.737330 12510 raft_consensus.cc:729] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.737350 12509 raft_consensus.cc:348] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.737370 12510 consensus_queue.cc:260] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.737422 12509 raft_consensus.cc:374] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.737442 12509 raft_consensus.cc:729] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.737461 12510 ts_tablet_manager.cc:1428] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:22.737479 12509 consensus_queue.cc:260] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.737557 12509 ts_tablet_manager.cc:1428] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.738970 12555 ts_tablet_manager.cc:1918] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:22.739051 12555 log.cc:1196] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/9ab143b2735f4ee2b5bec7a5fa120afe
I20260810 15:16:22.739307 12555 ts_tablet_manager.cc:1939] T 9ab143b2735f4ee2b5bec7a5fa120afe P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting consensus metadata
I20260810 15:16:22.739705 11625 catalog_manager.cc:4841] TS 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813): tablet 9ab143b2735f4ee2b5bec7a5fa120afe (table test-table [id=e4d2be5acadd4713962c214409b948e0]) successfully deleted
W20260810 15:16:22.765203 11989 tablet.cc:2369] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:16:22.833967 12460 tablet.cc:2369] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:22.926473 12521 raft_consensus.cc:482] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:22.926586 12521 raft_consensus.cc:504] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.926918 12521 leader_election.cc:288] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:22.929755 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.927231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:40341 (local address 127.29.230.1:45819)
0810 15:16:22.927302 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.927304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.927316 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:16:22.927350 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.927351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.927364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.927435 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.927604 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.928119 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929083 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.929438 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929500 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.929509 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.929659 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.929697 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:16:22.929755 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.927140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45819 (local address 127.29.230.3:40341)
0810 15:16:22.927206 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.927219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.927289 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.927444 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.927446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.927448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.927531 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.927533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.928204 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.928205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.928982 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.928984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.929523 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.929591 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.929594 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.929640 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.929681 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 15:16:22.929849 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.927365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:56575 (local address 127.29.230.2:36079)
0810 15:16:22.927438 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.927440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.927462 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:16:22.927509 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.927510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.927518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.927576 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:22.927739 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.928197 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929191 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.929532 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929598 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.929607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.929746 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.929782 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:16:22.929873 12563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.927242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36079 (local address 127.29.230.3:56575)
0810 15:16:22.927410 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.927419 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.927458 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.927590 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.927591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.927592 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:22.927670 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.927672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.928279 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.928280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.929107 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.929110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.929622 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.929623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.929681 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.929684 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.929729 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.929767 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260810 15:16:22.930024 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544feca7ece64998bd9e3d93d3387a00" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" is_pre_election: true
I20260810 15:16:22.930116 12148 raft_consensus.cc:2454] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 0.
I20260810 15:16:22.930058 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544feca7ece64998bd9e3d93d3387a00" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433" is_pre_election: true
I20260810 15:16:22.930188 11923 raft_consensus.cc:2454] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 0.
I20260810 15:16:22.930281 12267 leader_election.cc:302] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:22.930401 12521 raft_consensus.cc:2781] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:22.930434 12521 raft_consensus.cc:482] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:22.930446 12521 raft_consensus.cc:3037] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.931092 12521 raft_consensus.cc:504] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.931198 12521 leader_election.cc:288] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Requested vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:22.931437 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544feca7ece64998bd9e3d93d3387a00" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
I20260810 15:16:22.931429 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544feca7ece64998bd9e3d93d3387a00" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433"
I20260810 15:16:22.931504 12148 raft_consensus.cc:3037] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.931506 11923 raft_consensus.cc:3037] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:22.932075 12148 raft_consensus.cc:2454] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:22.932215 11923 raft_consensus.cc:2454] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:22.932250 12267 leader_election.cc:302] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:22.932325 12521 raft_consensus.cc:2781] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:22.932461 12521 raft_consensus.cc:686] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Becoming Leader. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Running, Role: LEADER
I20260810 15:16:22.932534 12521 consensus_queue.cc:237] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.933271 11649 catalog_manager.cc:5495] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3). New cstate: current_term: 1 leader_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } health_report { overall_health: HEALTHY } } }
Aug 10 15:16:22 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374982, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.62@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:22.986974 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.980047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:45377 (local address 127.0.0.1:38736)
0810 15:16:22.980226 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:22.980238 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:22.980305 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:22.980434 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:22.980435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:22.980767 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:22.980820 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.980822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.981531 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.981532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.982088 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:22.982090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.982488 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.982488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:22.982543 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.983107 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:22.983111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:22.984494 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:22.985962 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.985968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.985971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.986158 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.986324 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:22.986324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:22.986325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:22.986373 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:22.986604 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:22.986611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:22.986693 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 15:16:22.986764 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:16:22.986897 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260810 15:16:22.987063 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:22.980169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.29.230.62:45377)
0810 15:16:22.980239 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:22.980240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:22.980299 (+ 59us) server_negotiation.cc:407] Connection header received
0810 15:16:22.980366 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:22.980367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:22.980375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:22.980414 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:22.980900 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.981438 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.982198 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:22.982387 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:22.982450 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:22.984588 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:22.984590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:22.984590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:22.984597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:22.985877 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.986219 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.986220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.986220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.986271 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:22.986412 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:22.986412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:22.986413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:22.986521 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:22.986587 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:22.986812 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 15:16:22.986949 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":49}
I20260810 15:16:22.989004 11649 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:38736:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 15:16:22.989198 11649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:22.990653 12117 tablet_service.cc:1467] Processing CreateTablet for tablet d392edb8fa4f48a39e124e05c6de7eae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dc9915f237405b8d5530ec20b49476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:16:22.990660 11888 tablet_service.cc:1467] Processing CreateTablet for tablet d392edb8fa4f48a39e124e05c6de7eae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dc9915f237405b8d5530ec20b49476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:16:22.990679 12351 tablet_service.cc:1467] Processing CreateTablet for tablet d392edb8fa4f48a39e124e05c6de7eae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dc9915f237405b8d5530ec20b49476]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 15:16:22.990828 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d392edb8fa4f48a39e124e05c6de7eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.990830 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d392edb8fa4f48a39e124e05c6de7eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.990829 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d392edb8fa4f48a39e124e05c6de7eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:22.992233 12509 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: Bootstrap starting.
I20260810 15:16:22.992388 12520 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: Bootstrap starting.
I20260810 15:16:22.992659 12509 tablet_bootstrap.cc:654] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.992686 12520 tablet_bootstrap.cc:654] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.992925 12510 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: Bootstrap starting.
I20260810 15:16:22.993280 12510 tablet_bootstrap.cc:654] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:22.993299 12509 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: No bootstrap required, opened a new log
I20260810 15:16:22.993337 12509 ts_tablet_manager.cc:1397] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:22.993479 12509 raft_consensus.cc:348] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.993543 12509 raft_consensus.cc:374] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.993563 12509 raft_consensus.cc:729] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.993616 12509 consensus_queue.cc:260] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.993676 12520 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: No bootstrap required, opened a new log
I20260810 15:16:22.993712 12509 ts_tablet_manager.cc:1428] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.993721 12520 ts_tablet_manager.cc:1397] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:22.993820 12520 raft_consensus.cc:348] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.993882 12520 raft_consensus.cc:374] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.993899 12520 raft_consensus.cc:729] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027b619891646989611b2f614641433, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.993952 12520 consensus_queue.cc:260] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.994055 12520 ts_tablet_manager.cc:1428] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.994143 12526 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:22.994215 12526 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:22.994513 12510 tablet_bootstrap.cc:492] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: No bootstrap required, opened a new log
I20260810 15:16:22.994563 12510 ts_tablet_manager.cc:1397] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:16:22.994714 12510 raft_consensus.cc:348] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.994797 12510 raft_consensus.cc:374] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:22.994817 12510 raft_consensus.cc:729] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Initialized, Role: FOLLOWER
I20260810 15:16:22.994853 12510 consensus_queue.cc:260] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:22.994951 12510 ts_tablet_manager.cc:1428] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:22.995003 12523 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:22.995121 12523 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:22.995083 12521 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:22.995201 12521 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:23.130350 12523 raft_consensus.cc:482] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:23.130457 12523 raft_consensus.cc:504] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.130802 12523 leader_election.cc:288] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813)
I20260810 15:16:23.133704 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.131161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:52141 (local address 127.29.230.1:45819)
0810 15:16:23.131232 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.131235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.131251 (+ 16us) server_negotiation.cc:407] Connection header received
0810 15:16:23.131398 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.131399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.131408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.131456 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:23.131621 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.132094 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133045 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.133397 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133458 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.133467 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.133607 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.133643 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:16:23.133745 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.131328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:58843 (local address 127.29.230.3:39813)
0810 15:16:23.131397 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.131399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.131408 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:23.131437 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.131438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.131446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.131502 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:23.131663 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.132138 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133084 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.133421 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133489 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.133498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.133643 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.133679 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:16:23.133773 12581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.131027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45819 (local address 127.29.230.2:52141)
0810 15:16:23.131170 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.131182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:23.131301 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:23.131475 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:23.131476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:23.131478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:23.131551 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.131553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.132172 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.132173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.132947 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.132949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.133476 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.133541 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.133544 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:23.133590 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:16:23.133629 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260810 15:16:23.133916 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.131197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39813 (local address 127.29.230.2:58843)
0810 15:16:23.131319 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.131329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:23.131384 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:23.131522 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:23.131523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:23.131525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:23.131595 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.131597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.132226 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.132227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.132990 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.132992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.133507 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.133508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.133574 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.133578 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:23.133623 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:23.133663 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260810 15:16:23.133999 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d392edb8fa4f48a39e124e05c6de7eae" candidate_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433" is_pre_election: true
I20260810 15:16:23.134092 11923 raft_consensus.cc:2454] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9e3ef6a0c243f6a5432407d36add0b in term 0.
I20260810 15:16:23.134230 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d392edb8fa4f48a39e124e05c6de7eae" candidate_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" is_pre_election: true
I20260810 15:16:23.134325 12383 raft_consensus.cc:2454] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9e3ef6a0c243f6a5432407d36add0b in term 0.
I20260810 15:16:23.134320 12029 leader_election.cc:302] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4027b619891646989611b2f614641433; no voters:
I20260810 15:16:23.134449 12523 raft_consensus.cc:2781] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:23.134491 12523 raft_consensus.cc:482] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:23.134510 12523 raft_consensus.cc:3037] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.135167 12523 raft_consensus.cc:504] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.135301 12523 leader_election.cc:288] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [CANDIDATE]: Term 1 election: Requested vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813)
I20260810 15:16:23.135465 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d392edb8fa4f48a39e124e05c6de7eae" candidate_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433"
I20260810 15:16:23.135469 12383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d392edb8fa4f48a39e124e05c6de7eae" candidate_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f33600ce69b48a0a5b2d011199ebd0c"
I20260810 15:16:23.135532 11923 raft_consensus.cc:3037] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.135540 12383 raft_consensus.cc:3037] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.136085 11923 raft_consensus.cc:2454] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9e3ef6a0c243f6a5432407d36add0b in term 1.
I20260810 15:16:23.136214 12383 raft_consensus.cc:2454] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9e3ef6a0c243f6a5432407d36add0b in term 1.
I20260810 15:16:23.136245 12029 leader_election.cc:302] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4027b619891646989611b2f614641433; no voters:
I20260810 15:16:23.136356 12523 raft_consensus.cc:2781] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:23.136464 12523 raft_consensus.cc:686] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 LEADER]: Becoming Leader. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Running, Role: LEADER
I20260810 15:16:23.136525 12523 consensus_queue.cc:237] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.136864 12584 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: TxnStatusTablet state changed. Reason: New leader 0f9e3ef6a0c243f6a5432407d36add0b. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.136935 12584 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:16:23.137111 12587 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 15:16:23.137218 12587 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 15:16:23.137317 11651 catalog_manager.cc:5495] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2). New cstate: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } health_report { overall_health: UNKNOWN } } }
I20260810 15:16:23.143217 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.140791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.29.230.2:36079)
0810 15:16:23.140825 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.140827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.140871 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:16:23.140968 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.140968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.140987 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.141054 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:23.141199 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.141668 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.142490 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.142680 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.142735 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.142923 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.143019 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.143082 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.143138 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.143156 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:16:23.143253 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.140745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36079 (local address 127.0.0.1:42466)
0810 15:16:23.140826 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.140836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:23.140891 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:23.141085 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:23.141086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:23.141087 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:16:23.141137 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.141140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.141751 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.141752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.142396 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.142399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.142751 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.142752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.142822 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.142836 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.143083 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.143091 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 15:16:23.143135 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:16:23.143187 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 15:16:23.144378 12383 raft_consensus.cc:1261] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Refusing update from remote peer 0f9e3ef6a0c243f6a5432407d36add0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.144409 11923 raft_consensus.cc:1261] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Refusing update from remote peer 0f9e3ef6a0c243f6a5432407d36add0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.144726 12584 consensus_queue.cc:1035] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.144910 12523 consensus_queue.cc:1035] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.145454 12521 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: TxnStatusTablet state changed. Reason: New leader 0f9e3ef6a0c243f6a5432407d36add0b. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.145450 12526 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: TxnStatusTablet state changed. Reason: New leader 0f9e3ef6a0c243f6a5432407d36add0b. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.145548 12526 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:23.145548 12521 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:23.146481 12523 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.146476 12584 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.146570 12523 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:16:23.146580 12584 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 0f9e3ef6a0c243f6a5432407d36add0b: This TxnStatusTablet replica's current role is: LEADER
I20260810 15:16:23.146891 12526 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.146971 12526 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4027b619891646989611b2f614641433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:23.147632 12521 tablet_replica.cc:439] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } } }
I20260810 15:16:23.147720 12521 tablet_replica.cc:442] T d392edb8fa4f48a39e124e05c6de7eae P 4f33600ce69b48a0a5b2d011199ebd0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 15:16:23.151046 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.148370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.29.230.3:39813)
0810 15:16:23.148421 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.148423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.148460 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:16:23.148493 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.148494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.148515 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.148585 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:23.148664 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.149383 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.150159 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.150470 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.150536 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.150681 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.150806 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.150877 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.150955 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.150977 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:16:23.156651 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.153700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:59315 (local address 127.29.230.2:36079)
0810 15:16:23.153744 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.153746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.153846 (+ 100us) server_negotiation.cc:407] Connection header received
0810 15:16:23.153890 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.153891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.153901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.153950 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:23.154068 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.154781 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.155735 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.156054 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.156132 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.156398 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.156428 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.156484 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.156559 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.156583 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:16:23.156683 12598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.153602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36079 (local address 127.29.230.3:59315)
0810 15:16:23.153773 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.153783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:23.153838 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:23.153971 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:23.153972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:23.153973 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:16:23.154013 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.154015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.154873 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.154875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.155649 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.155652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.156201 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.156202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.156265 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.156280 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.156502 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.156504 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:23.156550 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:16:23.156593 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260810 15:16:23.159051 11923 raft_consensus.cc:1261] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.159360 12564 consensus_queue.cc:1035] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.161347 12148 raft_consensus.cc:1261] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.161623 12564 consensus_queue.cc:1035] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.170115 12605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.166521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39813 (local address 127.29.230.2:43539)
0810 15:16:23.166689 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.166700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:23.166752 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:23.166877 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:23.166878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:23.166879 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 15:16:23.166926 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.166928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.168398 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.168399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.169275 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:23.169277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.169732 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.169733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:23.169797 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.169810 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.169985 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.169986 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 15:16:23.170013 (+ 27us) client_negotiation.cc:240] Negotiation successful
0810 15:16:23.170047 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260810 15:16:23.170115 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.166555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:43539 (local address 127.29.230.3:39813)
0810 15:16:23.166595 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.166597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.166772 (+ 175us) server_negotiation.cc:407] Connection header received
0810 15:16:23.166800 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.166801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.166811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.166876 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:23.166964 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.167783 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.169349 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.169654 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.169723 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.169892 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.169927 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.169979 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.170026 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.170047 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:16:23.178642 11649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.178742 11649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.179358 11649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Sending DeleteTablet for 3 replicas of tablet 544feca7ece64998bd9e3d93d3387a00
I20260810 15:16:23.179713 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 544feca7ece64998bd9e3d93d3387a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260810 15:16:23.179716 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 544feca7ece64998bd9e3d93d3387a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:59254
I20260810 15:16:23.179813 12556 tablet_replica.cc:330] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: stopping tablet replica
I20260810 15:16:23.179817 12351 tablet_service.cc:1514] Processing DeleteTablet for tablet 544feca7ece64998bd9e3d93d3387a00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260810 15:16:23.179879 12556 raft_consensus.cc:2229] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.179896 12559 tablet_replica.cc:330] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: stopping tablet replica
I20260810 15:16:23.179932 12556 raft_consensus.cc:2258] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.179942 12559 raft_consensus.cc:2229] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.179973 12559 raft_consensus.cc:2258] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.180042 12555 tablet_replica.cc:330] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: stopping tablet replica
I20260810 15:16:23.180095 12555 raft_consensus.cc:2229] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:23.180181 12555 raft_consensus.cc:2258] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.180265 12559 ts_tablet_manager.cc:1905] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.180320 12556 ts_tablet_manager.cc:1905] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.180544 12555 ts_tablet_manager.cc:1905] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.182039 12559 ts_tablet_manager.cc:1918] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:16:23.182039 12556 ts_tablet_manager.cc:1918] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:16:23.182093 12559 log.cc:1196] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/544feca7ece64998bd9e3d93d3387a00
I20260810 15:16:23.182093 12556 log.cc:1196] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/544feca7ece64998bd9e3d93d3387a00
I20260810 15:16:23.182296 12556 ts_tablet_manager.cc:1939] T 544feca7ece64998bd9e3d93d3387a00 P 4027b619891646989611b2f614641433: Deleting consensus metadata
I20260810 15:16:23.182296 12559 ts_tablet_manager.cc:1939] T 544feca7ece64998bd9e3d93d3387a00 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting consensus metadata
I20260810 15:16:23.182462 12555 ts_tablet_manager.cc:1918] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:16:23.182538 12555 log.cc:1196] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/544feca7ece64998bd9e3d93d3387a00
I20260810 15:16:23.182720 12555 ts_tablet_manager.cc:1939] T 544feca7ece64998bd9e3d93d3387a00 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting consensus metadata
I20260810 15:16:23.182792 11625 catalog_manager.cc:4841] TS 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079): tablet 544feca7ece64998bd9e3d93d3387a00 (table test-table [id=42c5523a88694755b40aa5bcdce9e19e]) successfully deleted
I20260810 15:16:23.183136 11625 catalog_manager.cc:4841] TS 4027b619891646989611b2f614641433 (127.29.230.1:45819): tablet 544feca7ece64998bd9e3d93d3387a00 (table test-table [id=42c5523a88694755b40aa5bcdce9e19e]) successfully deleted
I20260810 15:16:23.185344 11625 catalog_manager.cc:4841] TS 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813): tablet 544feca7ece64998bd9e3d93d3387a00 (table test-table [id=42c5523a88694755b40aa5bcdce9e19e]) successfully deleted
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[11571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:16:23.217410 11756 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:23.210480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.29.230.62:45377)
0810 15:16:23.210533 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.210535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.210708 (+ 173us) server_negotiation.cc:407] Connection header received
0810 15:16:23.210749 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.210750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.210763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.210829 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:23.211636 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.212410 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.213540 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.213844 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.213923 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.217340 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38740: BlockingRecv error: recv got EOF from 127.0.0.1:38740 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[11571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM
I20260810 15:16:23.227381 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.218426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.29.230.62:45377)
0810 15:16:23.218467 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.218469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.218704 (+ 235us) server_negotiation.cc:407] Connection header received
0810 15:16:23.218737 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.218738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.218749 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.218814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:23.219362 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.220075 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.221109 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.221409 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.221487 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.224222 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:23.224224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:23.224224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:23.224233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:23.225950 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:23.226359 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:23.226360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:23.226361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:23.226420 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:23.226699 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:23.226700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:23.226700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:23.226854 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:23.226933 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.227169 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.227259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:16:23.229343 11649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:23.229559 11649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:23.231036 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad6d60480e48f69f2273697916d7f0 (DEFAULT_TABLE table=test-table [id=a9af64c86b174284ab290b8ff24ff5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.231127 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad6d60480e48f69f2273697916d7f0 (DEFAULT_TABLE table=test-table [id=a9af64c86b174284ab290b8ff24ff5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.231184 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad6d60480e48f69f2273697916d7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.231256 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad6d60480e48f69f2273697916d7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.232631 12351 tablet_service.cc:1467] Processing CreateTablet for tablet 98ad6d60480e48f69f2273697916d7f0 (DEFAULT_TABLE table=test-table [id=a9af64c86b174284ab290b8ff24ff5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.232656 12510 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Bootstrap starting.
I20260810 15:16:23.232745 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98ad6d60480e48f69f2273697916d7f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.233057 12510 tablet_bootstrap.cc:654] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.233982 12509 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Bootstrap starting.
I20260810 15:16:23.234021 12520 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Bootstrap starting.
I20260810 15:16:23.234361 12520 tablet_bootstrap.cc:654] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.234421 12509 tablet_bootstrap.cc:654] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.235250 12509 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: No bootstrap required, opened a new log
I20260810 15:16:23.235345 12509 ts_tablet_manager.cc:1397] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:23.235500 12509 raft_consensus.cc:348] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.235580 12509 raft_consensus.cc:374] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.235601 12509 raft_consensus.cc:729] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.235610 12520 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: No bootstrap required, opened a new log
I20260810 15:16:23.235652 12520 ts_tablet_manager.cc:1397] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:16:23.235643 12509 consensus_queue.cc:260] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.235731 12509 ts_tablet_manager.cc:1428] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:23.235749 12520 raft_consensus.cc:348] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.235810 12520 raft_consensus.cc:374] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.235827 12520 raft_consensus.cc:729] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027b619891646989611b2f614641433, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.235868 12520 consensus_queue.cc:260] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.235966 12520 ts_tablet_manager.cc:1428] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:23.236522 12510 tablet_bootstrap.cc:492] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: No bootstrap required, opened a new log
I20260810 15:16:23.236570 12510 ts_tablet_manager.cc:1397] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 15:16:23.236711 12510 raft_consensus.cc:348] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.236770 12510 raft_consensus.cc:374] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.236807 12510 raft_consensus.cc:729] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.236846 12510 consensus_queue.cc:260] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.236948 12510 ts_tablet_manager.cc:1428] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:23.427145 12564 raft_consensus.cc:482] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:23.427248 12564 raft_consensus.cc:504] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.427436 12564 leader_election.cc:288] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:23.427703 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad6d60480e48f69f2273697916d7f0" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433" is_pre_election: true
I20260810 15:16:23.427703 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad6d60480e48f69f2273697916d7f0" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" is_pre_election: true
I20260810 15:16:23.427804 12148 raft_consensus.cc:2454] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 0.
I20260810 15:16:23.427804 11923 raft_consensus.cc:2454] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 0.
I20260810 15:16:23.427958 12267 leader_election.cc:302] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:23.428072 12564 raft_consensus.cc:2781] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:23.428134 12564 raft_consensus.cc:482] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:23.428154 12564 raft_consensus.cc:3037] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.428877 12564 raft_consensus.cc:504] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.429014 12564 leader_election.cc:288] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Requested vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:23.429248 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad6d60480e48f69f2273697916d7f0" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
I20260810 15:16:23.429244 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98ad6d60480e48f69f2273697916d7f0" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433"
I20260810 15:16:23.429322 12148 raft_consensus.cc:3037] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.429322 11923 raft_consensus.cc:3037] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.429929 12148 raft_consensus.cc:2454] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:23.430050 11923 raft_consensus.cc:2454] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:23.430222 12267 leader_election.cc:302] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4027b619891646989611b2f614641433, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:23.430325 12564 raft_consensus.cc:2781] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:23.430380 12564 raft_consensus.cc:686] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Becoming Leader. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Running, Role: LEADER
I20260810 15:16:23.430431 12564 consensus_queue.cc:237] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.431035 11651 catalog_manager.cc:5495] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3). New cstate: current_term: 1 leader_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } health_report { overall_health: HEALTHY } } }
I20260810 15:16:23.465543 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.463345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.29.230.3:39813)
0810 15:16:23.463415 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:23.463417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:23.463427 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:23.463496 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:23.463497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:23.463507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:23.463548 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:23.463712 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.464173 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.464943 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:23.465131 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:23.465180 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:23.465357 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:23.465382 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:23.465415 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:23.465465 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 15:16:23.465483 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:16:23.466390 12148 raft_consensus.cc:1261] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.466502 11923 raft_consensus.cc:1261] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.466632 12564 consensus_queue.cc:1035] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.466759 12521 consensus_queue.cc:1035] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.469551 11649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.469635 11649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.470273 11649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Sending DeleteTablet for 3 replicas of tablet 98ad6d60480e48f69f2273697916d7f0
I20260810 15:16:23.470495 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ad6d60480e48f69f2273697916d7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:59254
I20260810 15:16:23.470537 12351 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ad6d60480e48f69f2273697916d7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260810 15:16:23.470635 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 98ad6d60480e48f69f2273697916d7f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260810 15:16:23.470654 12555 tablet_replica.cc:330] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: stopping tablet replica
I20260810 15:16:23.470723 12555 raft_consensus.cc:2229] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:23.470734 12559 tablet_replica.cc:330] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: stopping tablet replica
I20260810 15:16:23.470784 12555 raft_consensus.cc:2258] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.470785 12559 raft_consensus.cc:2229] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.470831 12559 raft_consensus.cc:2258] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.470865 12556 tablet_replica.cc:330] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: stopping tablet replica
I20260810 15:16:23.470932 12556 raft_consensus.cc:2229] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.470919 11649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 15:16:23.471019 12556 raft_consensus.cc:2258] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 15:16:23.471050 11649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:23.471146 12559 ts_tablet_manager.cc:1905] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.471359 12556 ts_tablet_manager.cc:1905] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.472205 12117 tablet_service.cc:1467] Processing CreateTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf (DEFAULT_TABLE table=test-table [id=3fa10c3bb6d94127b289fe6921d5f9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.472218 12351 tablet_service.cc:1467] Processing CreateTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf (DEFAULT_TABLE table=test-table [id=3fa10c3bb6d94127b289fe6921d5f9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.472317 12351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935d0d37dd475ba83f2d5c83caaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.472316 12117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935d0d37dd475ba83f2d5c83caaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.472991 12555 ts_tablet_manager.cc:1905] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.473014 12559 ts_tablet_manager.cc:1918] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:23.473060 12559 log.cc:1196] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/98ad6d60480e48f69f2273697916d7f0
I20260810 15:16:23.473045 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf (DEFAULT_TABLE table=test-table [id=3fa10c3bb6d94127b289fe6921d5f9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:23.473258 12559 ts_tablet_manager.cc:1939] T 98ad6d60480e48f69f2273697916d7f0 P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting consensus metadata
I20260810 15:16:23.473728 12509 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Bootstrap starting.
I20260810 15:16:23.473723 12510 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Bootstrap starting.
I20260810 15:16:23.474046 12509 tablet_bootstrap.cc:654] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.474042 12510 tablet_bootstrap.cc:654] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.474301 12555 ts_tablet_manager.cc:1918] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:23.474366 12555 log.cc:1196] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/98ad6d60480e48f69f2273697916d7f0
I20260810 15:16:23.474361 11625 catalog_manager.cc:4841] TS 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079): tablet 98ad6d60480e48f69f2273697916d7f0 (table test-table [id=a9af64c86b174284ab290b8ff24ff5ed]) successfully deleted
I20260810 15:16:23.474566 12555 ts_tablet_manager.cc:1939] T 98ad6d60480e48f69f2273697916d7f0 P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting consensus metadata
I20260810 15:16:23.474669 12510 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: No bootstrap required, opened a new log
I20260810 15:16:23.474684 12509 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: No bootstrap required, opened a new log
I20260810 15:16:23.474725 12509 ts_tablet_manager.cc:1397] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:23.474723 12510 ts_tablet_manager.cc:1397] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:23.474769 12556 ts_tablet_manager.cc:1918] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:23.474823 12556 log.cc:1196] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/98ad6d60480e48f69f2273697916d7f0
I20260810 15:16:23.474826 12510 raft_consensus.cc:348] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.474826 12509 raft_consensus.cc:348] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.474886 12510 raft_consensus.cc:374] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.474886 12509 raft_consensus.cc:374] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.474906 12509 raft_consensus.cc:729] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.474906 12510 raft_consensus.cc:729] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9e3ef6a0c243f6a5432407d36add0b, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.474947 12510 consensus_queue.cc:260] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.474947 12509 consensus_queue.cc:260] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.475024 12556 ts_tablet_manager.cc:1939] T 98ad6d60480e48f69f2273697916d7f0 P 4027b619891646989611b2f614641433: Deleting consensus metadata
I20260810 15:16:23.475031 12510 ts_tablet_manager.cc:1428] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:23.475040 12509 ts_tablet_manager.cc:1428] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:23.475095 12564 raft_consensus.cc:482] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:23.475121 12564 raft_consensus.cc:504] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.475216 12564 leader_election.cc:288] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:23.475250 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b935d0d37dd475ba83f2d5c83caaeaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:23.475386 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935d0d37dd475ba83f2d5c83caaeaf" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" is_pre_election: true
I20260810 15:16:23.475472 12148 raft_consensus.cc:2454] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 0.
I20260810 15:16:23.475728 12267 leader_election.cc:302] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:23.475816 12564 raft_consensus.cc:2781] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:23.475849 12564 raft_consensus.cc:482] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:23.475873 12564 raft_consensus.cc:3037] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.476496 12564 raft_consensus.cc:504] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.476548 12520 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: Bootstrap starting.
I20260810 15:16:23.476619 12564 leader_election.cc:288] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Requested vote from peers 4027b619891646989611b2f614641433 (127.29.230.1:45819), 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079)
I20260810 15:16:23.476655 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935d0d37dd475ba83f2d5c83caaeaf" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433" is_pre_election: true
I20260810 15:16:23.476795 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935d0d37dd475ba83f2d5c83caaeaf" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f9e3ef6a0c243f6a5432407d36add0b"
I20260810 15:16:23.476835 12520 tablet_bootstrap.cc:654] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:23.476862 12148 raft_consensus.cc:3037] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.477005 11625 catalog_manager.cc:4841] TS 4027b619891646989611b2f614641433 (127.29.230.1:45819): tablet 98ad6d60480e48f69f2273697916d7f0 (table test-table [id=a9af64c86b174284ab290b8ff24ff5ed]) successfully deleted
I20260810 15:16:23.477087 11625 catalog_manager.cc:4841] TS 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813): tablet 98ad6d60480e48f69f2273697916d7f0 (table test-table [id=a9af64c86b174284ab290b8ff24ff5ed]) successfully deleted
I20260810 15:16:23.477406 12520 tablet_bootstrap.cc:492] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: No bootstrap required, opened a new log
I20260810 15:16:23.477456 12520 ts_tablet_manager.cc:1397] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:23.477463 12148 raft_consensus.cc:2454] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:23.477540 12520 raft_consensus.cc:348] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.477604 12520 raft_consensus.cc:374] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:23.477598 12267 leader_election.cc:302] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9e3ef6a0c243f6a5432407d36add0b, 4f33600ce69b48a0a5b2d011199ebd0c; no voters:
I20260810 15:16:23.477625 12520 raft_consensus.cc:729] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027b619891646989611b2f614641433, State: Initialized, Role: FOLLOWER
I20260810 15:16:23.477653 12520 consensus_queue.cc:260] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.477741 12520 ts_tablet_manager.cc:1428] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260810 15:16:23.477734 12267 leader_election.cc:341] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4027b619891646989611b2f614641433 (127.29.230.1:45819): Illegal state: must be running to vote when last-logged opid is not known
I20260810 15:16:23.477865 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b935d0d37dd475ba83f2d5c83caaeaf" candidate_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4027b619891646989611b2f614641433"
I20260810 15:16:23.477919 11923 raft_consensus.cc:3037] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:23.478348 11923 raft_consensus.cc:2454] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f33600ce69b48a0a5b2d011199ebd0c in term 1.
I20260810 15:16:23.478636 12650 raft_consensus.cc:2781] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:23.478710 12650 raft_consensus.cc:686] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Becoming Leader. State: Replica: 4f33600ce69b48a0a5b2d011199ebd0c, State: Running, Role: LEADER
I20260810 15:16:23.478807 12650 consensus_queue.cc:237] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } }
I20260810 15:16:23.479310 11649 catalog_manager.cc:5495] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3). New cstate: current_term: 1 leader_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f33600ce69b48a0a5b2d011199ebd0c" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39813 } health_report { overall_health: HEALTHY } } }
I20260810 15:16:23.484862 12148 raft_consensus.cc:1261] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.484874 11923 raft_consensus.cc:1261] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Refusing update from remote peer 4f33600ce69b48a0a5b2d011199ebd0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:23.485051 12650 consensus_queue.cc:1035] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f9e3ef6a0c243f6a5432407d36add0b" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.485149 12564 consensus_queue.cc:1035] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4027b619891646989611b2f614641433" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:23.502103 11649 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.502213 11649 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:23.502903 11649 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 493745a1c58348ba81776d4a38260c74: Sending DeleteTablet for 3 replicas of tablet 0b935d0d37dd475ba83f2d5c83caaeaf
I20260810 15:16:23.503196 12117 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:42456
I20260810 15:16:23.503216 12351 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60408
I20260810 15:16:23.503197 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b935d0d37dd475ba83f2d5c83caaeaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:23 UTC) from {username='oryx', principal='oryx/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:59254
I20260810 15:16:23.503330 12556 tablet_replica.cc:330] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433: stopping tablet replica
I20260810 15:16:23.503328 12555 tablet_replica.cc:330] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: stopping tablet replica
I20260810 15:16:23.503355 12559 tablet_replica.cc:330] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: stopping tablet replica
I20260810 15:16:23.503386 12555 raft_consensus.cc:2229] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:16:23.503388 12556 raft_consensus.cc:2229] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.503417 12559 raft_consensus.cc:2229] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:23.503428 12556 raft_consensus.cc:2258] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4027b619891646989611b2f614641433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.503459 12559 raft_consensus.cc:2258] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.503476 12555 raft_consensus.cc:2258] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:23.503756 12555 ts_tablet_manager.cc:1905] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.503863 12559 ts_tablet_manager.cc:1905] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:23.503919 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 11782
I20260810 15:16:23.514935 12555 ts_tablet_manager.cc:1918] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:16:23.515019 12555 log.cc:1196] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/0b935d0d37dd475ba83f2d5c83caaeaf
I20260810 15:16:23.515240 12555 ts_tablet_manager.cc:1939] T 0b935d0d37dd475ba83f2d5c83caaeaf P 4f33600ce69b48a0a5b2d011199ebd0c: Deleting consensus metadata
I20260810 15:16:23.515736 11625 catalog_manager.cc:4841] TS 4f33600ce69b48a0a5b2d011199ebd0c (127.29.230.3:39813): tablet 0b935d0d37dd475ba83f2d5c83caaeaf (table test-table [id=3fa10c3bb6d94127b289fe6921d5f9ae]) successfully deleted
I20260810 15:16:23.517639 12559 ts_tablet_manager.cc:1918] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 15:16:23.517704 12559 log.cc:1196] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/0b935d0d37dd475ba83f2d5c83caaeaf
I20260810 15:16:23.517890 12559 ts_tablet_manager.cc:1939] T 0b935d0d37dd475ba83f2d5c83caaeaf P 0f9e3ef6a0c243f6a5432407d36add0b: Deleting consensus metadata
I20260810 15:16:23.518234 11625 catalog_manager.cc:4841] TS 0f9e3ef6a0c243f6a5432407d36add0b (127.29.230.2:36079): tablet 0b935d0d37dd475ba83f2d5c83caaeaf (table test-table [id=3fa10c3bb6d94127b289fe6921d5f9ae]) successfully deleted
W20260810 15:16:23.523442 11625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:45819 (error 108)
I20260810 15:16:23.524020 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 12008
I20260810 15:16:23.524075 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:23.523871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45819 (local address 127.0.0.1:59272)
0810 15:16:23.523990 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:23.524030 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
W20260810 15:16:23.534104 11625 catalog_manager.cc:4584] TS 4027b619891646989611b2f614641433 (127.29.230.1:45819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b935d0d37dd475ba83f2d5c83caaeaf: Network error: Client connection negotiation failed: client connection to 127.29.230.1:45819: connect: Connection refused (error 111)
I20260810 15:16:23.540239 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 12237
I20260810 15:16:23.552333 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 11601
2026-08-10T15:16:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2900 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[12670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[12670](info): set up 2 sockets
Aug 10 15:16:23 dist-test-slave-gjrz krb5kdc[12670](info): commencing operation
krb5kdc: starting...
W20260810 15:16:24.736219 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:24 dist-test-slave-gjrz krb5kdc[12670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:24.863502 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40051
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:24.946084 12699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:24.946305 12699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:24.946326 12699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:24.947728 12699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:24.947777 12699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:24.947790 12699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:24.947803 12699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:24.947813 12699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:24.949340 12699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:24.949604 12699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:24.949828 12699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:24.954877 12705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:24.954890 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:24.954877 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:24.955045 12699 server_base.cc:1034] running on GCE node
I20260810 15:16:24.955405 12699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:24.955708 12699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:24.956882 12699 hybrid_clock.cc:648] HybridClock initialized: now 1786374984956866 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T15:16:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1403 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:25 dist-test-slave-gjrz krb5kdc[12718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:25 dist-test-slave-gjrz krb5kdc[12718](info): set up 2 sockets
Aug 10 15:16:25 dist-test-slave-gjrz krb5kdc[12718](info): commencing operation
krb5kdc: starting...
W20260810 15:16:26.154471 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:26.276087 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46589
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:26.361364 12733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:26.361563 12733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:26.361584 12733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:26.362748 12733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:26.362792 12733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:26.362805 12733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:26.362816 12733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:26.362826 12733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:26.364351 12733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:26.364583 12733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:26.364811 12733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:26.369781 12740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.369803 12744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:26.369845 12733 server_base.cc:1034] running on GCE node
W20260810 15:16:26.369764 12739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:26.370208 12733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:26.370512 12733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:26.371665 12733 hybrid_clock.cc:648] HybridClock initialized: now 1786374986371644 us; error 37 us; skew 500 ppm
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:26.374132 12733 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:26.374543 12733 webserver.cc:460] Webserver started at http://127.29.230.62:45665/ using document root <none> and password file <none>
I20260810 15:16:26.374755 12733 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:26.374804 12733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:26.374934 12733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:26.376174 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "e876fe13070a4110899a6c98d1b91c74"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "d899bb614075c6a20baa4f965e615e1c"
server_key_iv: "bd2b19ad50b0b9f53f8c7ebebfb67923"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.376521 12733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "e876fe13070a4110899a6c98d1b91c74"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "d899bb614075c6a20baa4f965e615e1c"
server_key_iv: "bd2b19ad50b0b9f53f8c7ebebfb67923"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.377833 12733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:16:26.378538 12757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.378697 12733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:26.378791 12733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e876fe13070a4110899a6c98d1b91c74"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "d899bb614075c6a20baa4f965e615e1c"
server_key_iv: "bd2b19ad50b0b9f53f8c7ebebfb67923"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.378862 12733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:26.424625 12733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:26.426625 12733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:26.426769 12733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:26.430578 12733 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:46589
I20260810 15:16:26.430644 12850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:46589 every 8 connection(s)
I20260810 15:16:26.430958 12733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:26.431918 12851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:26.432209 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 12733
I20260810 15:16:26.432297 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:26.432474 30616 external_mini_cluster.cc:1428] Setting key f2b3914b6a5fec88218065bc744b7436
I20260810 15:16:26.434484 12851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Bootstrap starting.
I20260810 15:16:26.435066 12851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:26.435390 12851 log.cc:824] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:26.436173 12851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: No bootstrap required, opened a new log
I20260810 15:16:26.438513 12851 raft_consensus.cc:348] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:26.438687 12851 raft_consensus.cc:374] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:26.438722 12851 raft_consensus.cc:729] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e876fe13070a4110899a6c98d1b91c74, State: Initialized, Role: FOLLOWER
I20260810 15:16:26.438853 12851 consensus_queue.cc:260] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:26.438930 12851 raft_consensus.cc:388] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:26.438952 12851 raft_consensus.cc:482] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:26.439013 12851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:26.439623 12851 raft_consensus.cc:504] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:26.439724 12851 leader_election.cc:302] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e876fe13070a4110899a6c98d1b91c74; no voters:
I20260810 15:16:26.439891 12851 leader_election.cc:288] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:26.439951 12859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:26.440122 12859 raft_consensus.cc:686] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 LEADER]: Becoming Leader. State: Replica: e876fe13070a4110899a6c98d1b91c74, State: Running, Role: LEADER
I20260810 15:16:26.440168 12851 sys_catalog.cc:564] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:26.440234 12859 consensus_queue.cc:237] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:26.440703 12859 sys_catalog.cc:455] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e876fe13070a4110899a6c98d1b91c74. Latest consensus state: current_term: 1 leader_uuid: "e876fe13070a4110899a6c98d1b91c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } } }
I20260810 15:16:26.440791 12859 sys_catalog.cc:458] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:26.441056 12870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:26.441628 12870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:26.441699 12861 sys_catalog.cc:455] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e876fe13070a4110899a6c98d1b91c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } } }
I20260810 15:16:26.441825 12861 sys_catalog.cc:458] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:26.443320 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.433031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.29.230.62:46589)
0810 15:16:26.433206 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:26.433214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:26.433238 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:16:26.433644 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:26.433651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:26.433696 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:26.433820 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:26.434488 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.435588 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.436604 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.437005 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.437089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.439756 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:26.439760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:26.439760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:26.439770 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:26.441619 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.442045 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.442048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.442048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.442152 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.442398 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.442400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.442400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.442700 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:26.442839 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:26.442963 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:16:26.443109 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260810 15:16:26.447772 12870 catalog_manager.cc:1347] Generated new cluster ID: 13ba1625fa364c7e828d552e09604da2
I20260810 15:16:26.447829 12870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:26.462826 12870 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:26.463465 12870 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:26.481335 12870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Generated new TSK 0
I20260810 15:16:26.481541 12870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:26.511330 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:26.596055 12887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:26.596275 12887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:26.596312 12887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:26.597486 12887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:26.597525 12887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:26.597560 12887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:26.598990 12887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:26.599220 12887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:26.599457 12887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:26.604249 12894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.604259 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.604259 12899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:26.604598 12887 server_base.cc:1034] running on GCE node
I20260810 15:16:26.604775 12887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:26.604966 12887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:26.606099 12887 hybrid_clock.cc:648] HybridClock initialized: now 1786374986606090 us; error 28 us; skew 500 ppm
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:26.608611 12887 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:26.609010 12887 webserver.cc:460] Webserver started at http://127.29.230.1:35833/ using document root <none> and password file <none>
I20260810 15:16:26.609206 12887 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:26.609254 12887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:26.609382 12887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:26.610374 12887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "a96b094a25f64caba46b5eafbce3d3fa"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "ba9a41356f542ff0aa46c051e967f151"
server_key_iv: "9609eb6f7064c504073193ee24321736"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.610718 12887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "a96b094a25f64caba46b5eafbce3d3fa"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "ba9a41356f542ff0aa46c051e967f151"
server_key_iv: "9609eb6f7064c504073193ee24321736"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.612077 12887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:26.612859 12911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.613032 12887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:26.613122 12887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "a96b094a25f64caba46b5eafbce3d3fa"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "ba9a41356f542ff0aa46c051e967f151"
server_key_iv: "9609eb6f7064c504073193ee24321736"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.613188 12887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:26.637656 12887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:26.639799 12887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:26.639957 12887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:26.640237 12887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:26.640547 12887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:26.640583 12887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.640614 12887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:26.640630 12887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.646399 12887 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:39287
I20260810 15:16:26.646466 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:39287 every 8 connection(s)
I20260810 15:16:26.646761 12887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:26.647223 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 12887
I20260810 15:16:26.647353 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:26.647547 30616 external_mini_cluster.cc:1428] Setting key 90b06b1f457e05da806cea7bc34ddb7b
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:26.656719 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.647531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:50525 (local address 127.29.230.62:46589)
0810 15:16:26.647611 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:26.647613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:26.647840 (+ 227us) server_negotiation.cc:407] Connection header received
0810 15:16:26.648390 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:26.648391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:26.648401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:26.648437 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:26.649647 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.650288 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.651283 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.651554 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.651630 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.653922 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:26.653924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:26.653924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:26.653932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:26.655408 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.655808 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.655809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.655809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.655868 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.656085 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.656086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.656086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.656231 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:26.656303 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:26.656540 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 15:16:26.656630 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:16:26.656773 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.647511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:50525)
0810 15:16:26.647701 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:26.647755 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:26.648296 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:26.648523 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:26.648528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:26.649269 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:26.649541 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.649548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.650381 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.650382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.651193 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.651195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.651639 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.651640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.651747 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.652245 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:26.652252 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:26.653816 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:26.655499 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.655506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.655508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.655728 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.655931 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.655931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.655932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.656021 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.656293 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:26.656297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:26.656410 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 15:16:26.656487 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:16:26.656610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:16:26.657352 13111 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46589
I20260810 15:16:26.657437 13111 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:26.657625 13111 heartbeater.cc:511] Master 127.29.230.62:46589 requested a full tablet report, sending...
I20260810 15:16:26.658183 12787 ts_manager.cc:194] Registered new tserver with Master: a96b094a25f64caba46b5eafbce3d3fa (127.29.230.1:39287)
I20260810 15:16:26.658800 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:50525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:26.674039 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:26.756430 13118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:26.756639 13118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:26.756670 13118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:26.758145 13118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:26.758196 13118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:26.758232 13118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:26.760043 13118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:26.760314 13118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:26.760599 13118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:26.765803 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.765900 13132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.765807 13134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:26.766197 13118 server_base.cc:1034] running on GCE node
I20260810 15:16:26.766463 13118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:26.766762 13118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:26.767925 13118 hybrid_clock.cc:648] HybridClock initialized: now 1786374986767904 us; error 37 us; skew 500 ppm
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:26.770602 13118 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:26.771059 13118 webserver.cc:460] Webserver started at http://127.29.230.2:39711/ using document root <none> and password file <none>
I20260810 15:16:26.771256 13118 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:26.771343 13118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:26.771452 13118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:26.772478 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "170e77e3d6944b10b172ae1590d76ee0"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "7444d705715670ee768debc3368720a4"
server_key_iv: "04c49ea2b4ab5b5d8bfe1451aefe8ebd"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.772801 13118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "170e77e3d6944b10b172ae1590d76ee0"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "7444d705715670ee768debc3368720a4"
server_key_iv: "04c49ea2b4ab5b5d8bfe1451aefe8ebd"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.774075 13118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:16:26.774797 13144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.775005 13118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:26.775103 13118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "170e77e3d6944b10b172ae1590d76ee0"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "7444d705715670ee768debc3368720a4"
server_key_iv: "04c49ea2b4ab5b5d8bfe1451aefe8ebd"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.775169 13118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:26.792040 13118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:26.793782 13118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:26.793902 13118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:26.794099 13118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:26.794422 13118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:26.794461 13118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.794481 13118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:26.794492 13118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.800043 13118 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:36109
I20260810 15:16:26.800117 13350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:36109 every 8 connection(s)
I20260810 15:16:26.800390 13118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:26.809335 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 13118
I20260810 15:16:26.809391 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.801094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:43727 (local address 127.29.230.62:46589)
0810 15:16:26.801164 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:26.801166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:26.801405 (+ 239us) server_negotiation.cc:407] Connection header received
0810 15:16:26.801947 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:26.801948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:26.801960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:26.802015 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:26.802890 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.803473 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.804478 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.804667 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.804717 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.806728 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:26.806729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:26.806730 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:26.806739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:26.808176 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.808551 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.808551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.808552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.808600 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.808790 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.808790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.808791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.808885 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:26.808928 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:26.809209 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 15:16:26.809323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:16:26.809458 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:26.809552 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.801097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:43727)
0810 15:16:26.801285 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:26.801321 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:26.801787 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:26.802070 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:26.802073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:26.802616 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:26.802804 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.802808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.803581 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.803582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.804393 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.804396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.804736 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.804737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.804834 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.805296 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:26.805304 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:26.806622 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:26.808264 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.808272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.808273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.808487 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.808652 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.808653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.808653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.808744 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.808930 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:26.808933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:26.809045 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 15:16:26.809146 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 15:16:26.809313 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260810 15:16:26.809666 30616 external_mini_cluster.cc:1428] Setting key 5e6efd2f5b7c5ac45ca7c1e91cad0a8e
I20260810 15:16:26.810029 13352 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46589
I20260810 15:16:26.810122 13352 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:26.810328 13352 heartbeater.cc:511] Master 127.29.230.62:46589 requested a full tablet report, sending...
I20260810 15:16:26.810714 12787 ts_manager.cc:194] Registered new tserver with Master: 170e77e3d6944b10b172ae1590d76ee0 (127.29.230.2:36109)
I20260810 15:16:26.811081 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:43727
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:26.835892 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:26.918176 13359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:26.918377 13359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:26.918399 13359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:26.919540 13359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:26.919579 13359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:26.919602 13359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:26.921046 13359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:26.921238 13359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:26.921428 13359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:26.926512 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.926507 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:26.926597 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:26.926736 13359 server_base.cc:1034] running on GCE node
I20260810 15:16:26.926965 13359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:26.927215 13359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:26.928380 13359 hybrid_clock.cc:648] HybridClock initialized: now 1786374986928366 us; error 30 us; skew 500 ppm
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:26.930927 13359 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:26.931407 13359 webserver.cc:460] Webserver started at http://127.29.230.3:33627/ using document root <none> and password file <none>
I20260810 15:16:26.931630 13359 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:26.931681 13359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:26.931807 13359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:26.932822 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "bda67a52f8984810bd94af67540b7555"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "92b002e451faf1e96d121dd210e02234"
server_key_iv: "35d9ec96ea8d904576c7fe6608199dc8"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.933167 13359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "bda67a52f8984810bd94af67540b7555"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "92b002e451faf1e96d121dd210e02234"
server_key_iv: "35d9ec96ea8d904576c7fe6608199dc8"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.934433 13359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:26.935155 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.935354 13359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:26.935443 13359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "bda67a52f8984810bd94af67540b7555"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "92b002e451faf1e96d121dd210e02234"
server_key_iv: "35d9ec96ea8d904576c7fe6608199dc8"
server_key_version: "encryptionkey@0"
I20260810 15:16:26.935511 13359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:26.948753 13359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:26.950621 13359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:26.950760 13359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:26.950965 13359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:26.951315 13359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:26.951359 13359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.951380 13359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:26.951390 13359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:26.956851 13359 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:45211
I20260810 15:16:26.956943 13574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:45211 every 8 connection(s)
I20260810 15:16:26.957199 13359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:26.961303 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 13359
I20260810 15:16:26.961406 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:26.961588 30616 external_mini_cluster.cc:1428] Setting key b89a28ce7bd0dbc3473837f83aca081e
Aug 10 15:16:26 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:26.966828 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.957944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:36489 (local address 127.29.230.62:46589)
0810 15:16:26.958014 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:26.958016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:26.958250 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:16:26.958751 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:26.958752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:26.958766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:26.958819 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:26.959820 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.960330 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.961440 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.961701 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.961773 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.964285 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:26.964287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:26.964287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:26.964294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:26.965529 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.965884 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.965885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.965885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.965933 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:26.966234 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:26.966234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:26.966235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:26.966349 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:26.966419 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:26.966690 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 15:16:26.966763 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:16:26.966920 13580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:26.957902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:36489)
0810 15:16:26.958124 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:26.958160 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:26.958649 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:26.958887 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:26.958890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:26.959502 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:26.959717 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.959722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.960428 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.960429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.961329 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:26.961333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:26.961798 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:26.961800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:26.961973 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:26.962635 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:26.962646 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:26.964195 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:26.965609 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.965615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.965617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.965825 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.966050 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:26.966051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:26.966051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:26.966164 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:26.966412 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:26.966415 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:26.966532 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 15:16:26.966615 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 15:16:26.966745 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":128,"threads_started":1}
I20260810 15:16:26.967434 13575 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46589
I20260810 15:16:26.967530 13575 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:26.967718 13575 heartbeater.cc:511] Master 127.29.230.62:46589 requested a full tablet report, sending...
I20260810 15:16:26.968077 12787 ts_manager.cc:194] Registered new tserver with Master: bda67a52f8984810bd94af67540b7555 (127.29.230.3:45211)
I20260810 15:16:26.968436 12787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:36489
I20260810 15:16:26.973359 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
Entry for principal oryx/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab.
WARNING: no policy specified for oryx/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
Entry for principal oryx/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab.
WARNING: no policy specified for oryx/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
Entry for principal oryx/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab.
I20260810 15:16:27.047444 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 12887
I20260810 15:16:27.060191 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 13118
I20260810 15:16:27.070923 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 13359
I20260810 15:16:27.079493 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 12733
I20260810 15:16:27.093591 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46589
--webserver_interface=127.29.230.62
--webserver_port=45665
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:27.176577 13591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:27.176831 13591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:27.176868 13591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:27.178082 13591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:27.178130 13591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:27.178150 13591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:27.178166 13591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:27.178180 13591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:27.179733 13591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=45665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:27.179984 13591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:27.180213 13591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:27.185228 13606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.185256 13603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:27.185379 13591 server_base.cc:1034] running on GCE node
W20260810 15:16:27.185256 13599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:27.185709 13591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:27.185992 13591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:27.187139 13591 hybrid_clock.cc:648] HybridClock initialized: now 1786374987187120 us; error 36 us; skew 500 ppm
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:27.192804 13591 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:27.193236 13591 webserver.cc:460] Webserver started at http://127.29.230.62:45665/ using document root <none> and password file <none>
I20260810 15:16:27.193446 13591 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:27.193513 13591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:27.194885 13591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:27.195567 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.195750 13591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:27.195842 13591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e876fe13070a4110899a6c98d1b91c74"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "d899bb614075c6a20baa4f965e615e1c"
server_key_iv: "bd2b19ad50b0b9f53f8c7ebebfb67923"
server_key_version: "encryptionkey@0"
I20260810 15:16:27.196192 13591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:27.204483 13591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:27.206336 13591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:27.206470 13591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:27.210475 13591 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:46589
I20260810 15:16:27.210521 13700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:46589 every 8 connection(s)
I20260810 15:16:27.210889 13591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:27.212448 13702 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:16:27.213128 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Bootstrap starting.
I20260810 15:16:27.218775 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 13591
I20260810 15:16:27.219050 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:39287
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:46589
--webserver_port=35833
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:16:27.221510 13702 log.cc:824] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:27.223309 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:16:27.223600 13702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Bootstrap complete.
I20260810 15:16:27.225754 13702 raft_consensus.cc:348] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:27.225979 13702 raft_consensus.cc:729] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e876fe13070a4110899a6c98d1b91c74, State: Initialized, Role: FOLLOWER
I20260810 15:16:27.226136 13702 consensus_queue.cc:260] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:27.226214 13702 raft_consensus.cc:388] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:27.226263 13702 raft_consensus.cc:482] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:27.226305 13702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:16:27.227370 13702 raft_consensus.cc:504] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:27.227463 13702 leader_election.cc:302] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e876fe13070a4110899a6c98d1b91c74; no voters:
I20260810 15:16:27.227626 13702 leader_election.cc:288] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:16:27.227666 13710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:16:27.227826 13710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [term 2 LEADER]: Becoming Leader. State: Replica: e876fe13070a4110899a6c98d1b91c74, State: Running, Role: LEADER
I20260810 15:16:27.227905 13702 sys_catalog.cc:564] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:27.228040 13710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } }
I20260810 15:16:27.228693 13712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e876fe13070a4110899a6c98d1b91c74. Latest consensus state: current_term: 2 leader_uuid: "e876fe13070a4110899a6c98d1b91c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } } }
I20260810 15:16:27.228780 13712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:27.228714 13713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e876fe13070a4110899a6c98d1b91c74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e876fe13070a4110899a6c98d1b91c74" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46589 } } }
I20260810 15:16:27.228878 13713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:27.229040 13733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:27.229558 13733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:27.229689 13733 catalog_manager.cc:1259] Loaded cluster ID: 13ba1625fa364c7e828d552e09604da2
I20260810 15:16:27.229766 13733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:27.231127 13733 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:27.231317 13733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e876fe13070a4110899a6c98d1b91c74: Loaded TSK: 0
I20260810 15:16:27.231539 13733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:16:27.303668 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:27.303864 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:27.303896 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:27.305119 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:27.305173 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:27.305210 13706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:27.306682 13706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.1.keytab
--principal=oryx/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:39287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=35833
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:27.306911 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:27.307142 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:27.312160 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.312137 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.312160 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:27.312354 13706 server_base.cc:1034] running on GCE node
I20260810 15:16:27.312584 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:27.312819 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:27.313978 13706 hybrid_clock.cc:648] HybridClock initialized: now 1786374987313958 us; error 37 us; skew 500 ppm
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374987, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:27.316776 13706 init.cc:377] Logged in from keytab as oryx/127.29.230.1@KRBTEST.COM (short username oryx)
I20260810 15:16:27.317193 13706 webserver.cc:460] Webserver started at http://127.29.230.1:35833/ using document root <none> and password file <none>
I20260810 15:16:27.317422 13706 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:27.317485 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:27.318851 13706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:27.319517 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.319763 13706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:27.319864 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "a96b094a25f64caba46b5eafbce3d3fa"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "ba9a41356f542ff0aa46c051e967f151"
server_key_iv: "9609eb6f7064c504073193ee24321736"
server_key_version: "encryptionkey@0"
I20260810 15:16:27.320205 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:27.331174 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:27.333649 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:27.333791 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:27.334022 13706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:27.334362 13706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:27.334403 13706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.334432 13706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:27.334460 13706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.340293 13706 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:39287
I20260810 15:16:27.340361 13940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:39287 every 8 connection(s)
I20260810 15:16:27.340696 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:27.344887 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 13706
I20260810 15:16:27.345160 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:36109
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:46589
--webserver_port=39711
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.350478 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.341413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:50245)
0810 15:16:27.341595 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.341640 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.342253 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.342874 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.342877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.343463 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.343667 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.343672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.345307 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.345308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.346545 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.346548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.347383 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.347385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.347534 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.348048 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.348057 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.350280 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
W20260810 15:16:27.350768 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.341534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:50245 (local address 127.29.230.62:46589)
0810 15:16:27.342453 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.342463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.342498 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:16:27.342636 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.342642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.342692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.342815 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.343782 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.345186 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.346660 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.347259 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.347388 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.350435 (+ 3047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:50245: BlockingRecv error: recv got EOF from 127.29.230.1:50245 (error 108)
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":76,"threads_started":1}
W20260810 15:16:27.350795 13944 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:46589 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.230.62:46589: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.355065 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.351038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:35381)
0810 15:16:27.351143 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.351153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.351204 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.351339 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.351339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.351468 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.351511 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.351513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.352118 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.352118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.352623 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.352625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.352955 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.352956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.353019 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.353402 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.353407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.354997 (+ 1590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 15:16:27.355242 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.351099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:35381 (local address 127.29.230.62:46589)
0810 15:16:27.351132 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.351134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.351189 (+ 55us) server_negotiation.cc:407] Connection header received
0810 15:16:27.351250 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.351251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.351282 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.351335 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.351586 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.352039 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.352705 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.352889 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.352943 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.355105 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:35381: BlockingRecv error: recv got EOF from 127.29.230.1:35381 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.359163 13946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.355415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:44001)
0810 15:16:27.355458 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.355468 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.355499 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.355645 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.355647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.355761 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.355788 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.355789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.356383 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.356384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.356864 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.356866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.357206 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.357207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.357257 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.357616 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.357621 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.359102 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 15:16:27.359360 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.355475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:44001 (local address 127.29.230.62:46589)
0810 15:16:27.355529 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.355531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.355540 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:27.355561 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.355561 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.355569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.355621 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.355860 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.356306 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.356943 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.357139 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.357193 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.359196 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:44001: BlockingRecv error: recv got EOF from 127.29.230.1:44001 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 15:16:27.359385 13944 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:16:27.431620 13948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:27.431818 13948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:27.431839 13948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:27.433009 13948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:27.433053 13948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:27.433077 13948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:27.434520 13948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.2.keytab
--principal=oryx/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:36109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=39711
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:27.434739 13948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:27.434960 13948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:27.440019 13954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.440229 13955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.440243 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:27.440397 13948 server_base.cc:1034] running on GCE node
I20260810 15:16:27.440632 13948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:27.440878 13948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:27.442037 13948 hybrid_clock.cc:648] HybridClock initialized: now 1786374987442003 us; error 50 us; skew 500 ppm
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374987, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:27.444541 13948 init.cc:377] Logged in from keytab as oryx/127.29.230.2@KRBTEST.COM (short username oryx)
I20260810 15:16:27.444993 13948 webserver.cc:460] Webserver started at http://127.29.230.2:39711/ using document root <none> and password file <none>
I20260810 15:16:27.445215 13948 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:27.445281 13948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:27.446666 13948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:27.447402 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.447606 13948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:27.447698 13948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "170e77e3d6944b10b172ae1590d76ee0"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "7444d705715670ee768debc3368720a4"
server_key_iv: "04c49ea2b4ab5b5d8bfe1451aefe8ebd"
server_key_version: "encryptionkey@0"
I20260810 15:16:27.448038 13948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:27.478664 13948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:27.480825 13948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:27.480970 13948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:27.481200 13948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:27.481547 13948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:27.481585 13948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.481614 13948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:27.481639 13948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.487212 13948 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:36109
I20260810 15:16:27.487360 14162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:36109 every 8 connection(s)
I20260810 15:16:27.487644 13948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:27.492268 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 13948
I20260810 15:16:27.492584 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:45211
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:46589
--webserver_port=33627
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.496570 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.488301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:40777)
0810 15:16:27.488448 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.488505 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.488974 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.489312 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.489316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.489907 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.490114 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.490120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.490884 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.490886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.492234 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.492237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.492891 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.492892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.493064 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.493613 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.493624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.496366 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
W20260810 15:16:27.496629 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.488340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:40777 (local address 127.29.230.62:46589)
0810 15:16:27.488425 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.488427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.488573 (+ 146us) server_negotiation.cc:407] Connection header received
0810 15:16:27.489106 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.489108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.489123 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.489179 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.490225 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.490772 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.492323 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.492740 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.492832 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.496575 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:40777: BlockingRecv error: recv got EOF from 127.29.230.2:40777 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 15:16:27.496840 14163 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:46589 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.230.62:46589: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.501610 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.497051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:49573)
0810 15:16:27.497095 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.497104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.497139 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.497291 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.497292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.497421 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.497514 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.497517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.498358 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.498359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.498893 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.498894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.499234 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.499235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.499329 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.499737 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.499741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.501547 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:16:27.501695 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.497125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:49573 (local address 127.29.230.62:46589)
0810 15:16:27.497171 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.497173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.497179 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:16:27.497202 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.497203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.497212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.497257 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.497603 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.498281 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.498976 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.499168 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.499221 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.501645 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:49573: BlockingRecv error: recv got EOF from 127.29.230.2:49573 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.506026 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.502012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:59311)
0810 15:16:27.502071 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.502081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.502124 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.502249 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.502249 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.502368 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.502393 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.502395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.502991 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.502992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.503550 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.503552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.503865 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.503866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.503916 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.504329 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.504333 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.505964 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 15:16:27.506120 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.502094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:59311 (local address 127.29.230.62:46589)
0810 15:16:27.502151 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.502153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.502160 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:27.502180 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.502180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.502189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.502233 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.502472 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.502915 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.503626 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.503801 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.503853 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.506068 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:59311: BlockingRecv error: recv got EOF from 127.29.230.2:59311 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 15:16:27.506295 14163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:16:27.579677 14170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:27.579893 14170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:27.579924 14170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:27.581101 14170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:27.581143 14170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:27.581178 14170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:27.582732 14170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/krb5kdc/oryx_127.29.230.3.keytab
--principal=oryx/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:45211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=33627
--tserver_master_addrs=127.29.230.62:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:27.582980 14170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:27.583216 14170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:27.588308 14180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.588307 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:27.588307 14179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:27.588474 14170 server_base.cc:1034] running on GCE node
I20260810 15:16:27.588757 14170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:27.589000 14170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:27.590174 14170 hybrid_clock.cc:648] HybridClock initialized: now 1786374987590125 us; error 68 us; skew 500 ppm
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786374987, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:27.592921 14170 init.cc:377] Logged in from keytab as oryx/127.29.230.3@KRBTEST.COM (short username oryx)
I20260810 15:16:27.593364 14170 webserver.cc:460] Webserver started at http://127.29.230.3:33627/ using document root <none> and password file <none>
I20260810 15:16:27.593602 14170 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:27.593667 14170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:27.595105 14170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:27.595815 14196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.596009 14170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:27.596096 14170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "bda67a52f8984810bd94af67540b7555"
format_stamp: "Formatted at 2026-08-10 15:16:26 on dist-test-slave-gjrz"
server_key: "92b002e451faf1e96d121dd210e02234"
server_key_iv: "35d9ec96ea8d904576c7fe6608199dc8"
server_key_version: "encryptionkey@0"
I20260810 15:16:27.596436 14170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:27.607717 14170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:27.609797 14170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:27.609910 14170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:27.610116 14170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:27.610379 14170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:27.610412 14170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.610438 14170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:27.610455 14170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:27.615996 14170 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:45211
I20260810 15:16:27.616051 14383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:45211 every 8 connection(s)
I20260810 15:16:27.616371 14170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:27.620035 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 14170
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
W20260810 15:16:27.624178 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.617145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:45349 (local address 127.29.230.62:46589)
0810 15:16:27.617219 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.617221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.617384 (+ 163us) server_negotiation.cc:407] Connection header received
0810 15:16:27.617841 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.617842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.617851 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.617904 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.618824 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.619341 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.620398 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.620651 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.620721 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.624124 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:45349: BlockingRecv error: recv got EOF from 127.29.230.3:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:16:27.624228 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.617085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:45349)
0810 15:16:27.617267 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.617302 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.617767 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.617957 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.617960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.618527 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.618724 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.618730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.619431 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.619433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.620305 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.620308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.620743 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.620745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.620893 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.621465 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.621475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.624029 (+ 2554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260810 15:16:27.624521 14385 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:46589 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.230.62:46589: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
I20260810 15:16:27.627791 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.620542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.29.230.62:46589)
0810 15:16:27.620657 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.620658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.620665 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:16:27.620687 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.620688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.620699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.620771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.621399 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.622159 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.623109 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.623374 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.623441 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.624646 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:27.624651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:27.624652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:27.624664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:27.626362 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:27.626748 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:27.626751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:27.626751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:27.626870 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:27.627089 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:27.627090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:27.627090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:27.627372 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:27.627492 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:27.627632 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:16:27.627716 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.629434 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.624782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:39957)
0810 15:16:27.624827 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.624838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.624898 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.625013 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.625014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.625161 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.625208 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.625210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.626030 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.626031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.626586 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.626588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.626980 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.626981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.627052 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.627453 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.627458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.629391 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:16:27.629545 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.624839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:39957 (local address 127.29.230.62:46589)
0810 15:16:27.624882 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.624884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.624893 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:27.624948 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.624948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.624957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.624999 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.625284 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.625947 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.626666 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.626908 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.626966 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.629477 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:39957: BlockingRecv error: recv got EOF from 127.29.230.3:39957 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:27 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:27.633364 14391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:27.629735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:57325)
0810 15:16:27.629782 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:27.629792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:27.629830 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:27.629935 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:27.629935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:27.630074 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:27.630100 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.630102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.630685 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.630686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.631169 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:27.631170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.631532 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.631533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:27.631584 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.631870 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:27.631875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:27.633310 (+ 1435us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:16:27.633447 13947 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:27.629798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:57325 (local address 127.29.230.62:46589)
0810 15:16:27.629853 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:27.629854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:27.629862 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:27.629880 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:27.629881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:27.629888 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:27.629933 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:27.630175 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.630612 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.631247 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:27.631464 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:27.631527 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:27.633401 (+ 1874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:57325: BlockingRecv error: recv got EOF from 127.29.230.3:57325 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 15:16:27.633496 14385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:28.364396 14394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:28.359748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:49113)
0810 15:16:28.359880 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:28.359891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:28.359948 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:28.360113 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:28.360114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:28.360284 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:28.360337 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.360338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.360997 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.360998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.361558 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.361559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.362014 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.362015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.362078 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.362482 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:28.362487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:28.364326 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
W20260810 15:16:28.364559 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:28.359826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:49113 (local address 127.29.230.62:46589)
0810 15:16:28.359956 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:28.359957 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:28.359965 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:28.359990 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:28.359990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:28.360000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:28.360062 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:28.360401 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.360918 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.361671 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.361871 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.361999 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.364480 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:49113: BlockingRecv error: recv got EOF from 127.29.230.1:49113 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:28.511680 14396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:28.506715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:46129)
0810 15:16:28.506833 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:28.506845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:28.506904 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:28.507024 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:28.507025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:28.507228 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:28.507309 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.507311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.508009 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.508011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.508601 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.508602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.509085 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.509087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.509151 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.509599 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:28.509605 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:28.511585 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":68,"threads_started":1}
W20260810 15:16:28.511747 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:28.506816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:46129 (local address 127.29.230.62:46589)
0810 15:16:28.506898 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:28.506900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:28.506909 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:28.506963 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:28.506963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:28.506976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:28.507018 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:28.507418 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.507922 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.508778 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.508999 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.509069 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.511694 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:46129: BlockingRecv error: recv got EOF from 127.29.230.2:46129 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:28 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:28.638496 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:28.633862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:51385)
0810 15:16:28.634013 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:28.634025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:28.634097 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:28.634235 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:28.634236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:28.634414 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:28.634468 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.634470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.635156 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.635157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.635765 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:28.635768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.636142 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.636143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:28.636206 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.636626 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:28.636630 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:28.638408 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260810 15:16:28.638582 14395 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:28.633945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:51385 (local address 127.29.230.62:46589)
0810 15:16:28.634009 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:28.634011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:28.634091 (+ 80us) server_negotiation.cc:407] Connection header received
0810 15:16:28.634149 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:28.634150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:28.634164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:28.634215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:28.634557 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.635059 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.635856 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:28.636062 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:28.636121 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:28.638527 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:51385: BlockingRecv error: recv got EOF from 127.29.230.3:51385 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:29.369720 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:29.364879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:48567)
0810 15:16:29.365022 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:29.365033 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:29.365111 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:29.365305 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:29.365306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:29.365470 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:29.365523 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.365525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.366260 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.366261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.366910 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.366913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.367300 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.367301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.367362 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.367844 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:29.367850 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:29.369657 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
W20260810 15:16:29.369845 14402 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:29.365019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:48567 (local address 127.29.230.62:46589)
0810 15:16:29.365145 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:29.365146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:29.365154 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:29.365181 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:29.365182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:29.365192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:29.365286 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:29.365604 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.366146 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.366998 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.367198 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.367254 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.369767 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:48567: BlockingRecv error: recv got EOF from 127.29.230.1:48567 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:29.516687 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:29.512185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:39503)
0810 15:16:29.512301 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:29.512315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:29.512386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:29.512531 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:29.512532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:29.512705 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:29.512755 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.512757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.513419 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.513420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.513938 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.513940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.514286 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.514287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.514343 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.514778 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:29.514783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:29.516612 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
W20260810 15:16:29.516765 14402 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:29.512288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:39503 (local address 127.29.230.62:46589)
0810 15:16:29.512366 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:29.512368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:29.512378 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:29.512449 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:29.512450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:29.512463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:29.512510 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:29.512834 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.513333 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.514019 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.514216 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.514272 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.516712 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:39503: BlockingRecv error: recv got EOF from 127.29.230.2:39503 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:29 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:29.643623 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:29.638956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:45461)
0810 15:16:29.639105 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:29.639119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:29.639191 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:29.639321 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:29.639322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:29.639489 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:29.639542 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.639543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.640211 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.640212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.640780 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:29.640783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.641146 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.641148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:29.641202 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.641622 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:29.641627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:29.643558 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
W20260810 15:16:29.643713 14402 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:29.639090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:45461 (local address 127.29.230.62:46589)
0810 15:16:29.639143 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:29.639145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:29.639174 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:16:29.639239 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:29.639240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:29.639254 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:29.639315 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:29.639604 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.640112 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.640861 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:29.641056 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:29.641108 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:29.643652 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:45461: BlockingRecv error: recv got EOF from 127.29.230.3:45461 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:30.375329 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:30.370245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:47201)
0810 15:16:30.370412 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:30.370425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:30.370485 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:30.370754 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:30.370756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:30.370931 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:30.370984 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.370985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.371709 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.371711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.372540 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.372542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.372882 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.372882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.372946 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.373523 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:30.373529 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:30.375225 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
W20260810 15:16:30.375433 14407 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:30.370450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47201 (local address 127.29.230.62:46589)
0810 15:16:30.370610 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:30.370611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:30.370619 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:30.370642 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:30.370642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:30.370653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:30.370712 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:30.371052 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.371613 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.372618 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.372821 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.372875 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.375366 (+ 2491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:47201: BlockingRecv error: recv got EOF from 127.29.230.1:47201 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:30.522053 14409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:30.517236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:49535)
0810 15:16:30.517410 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:30.517422 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:30.517490 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:30.517663 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:30.517664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:30.517832 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:30.517885 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.517887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.518570 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.518571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.519108 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.519109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.519556 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.519557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.519616 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.520175 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:30.520179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:30.521990 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
W20260810 15:16:30.522182 14407 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:30.517356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:49535 (local address 127.29.230.62:46589)
0810 15:16:30.517431 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:30.517433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:30.517472 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:16:30.517552 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:30.517552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:30.517563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:30.517606 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:30.517963 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.518492 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.519204 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.519453 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.519512 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.522120 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:49535: BlockingRecv error: recv got EOF from 127.29.230.2:49535 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:30 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:30.648919 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:30.644151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:53389)
0810 15:16:30.644286 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:30.644296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:30.644349 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:30.644470 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:30.644471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:30.644642 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:30.644701 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.644704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.645472 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.645472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.646027 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:30.646030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.646438 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.646439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:30.646483 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.646919 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:30.646924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:30.648841 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260810 15:16:30.648998 14407 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:30.644254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:53389 (local address 127.29.230.62:46589)
0810 15:16:30.644333 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:30.644335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:30.644345 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:30.644394 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:30.644394 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:30.644406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:30.644456 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:30.644777 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.645414 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.646169 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:30.646365 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:30.646427 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:30.648944 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:53389: BlockingRecv error: recv got EOF from 127.29.230.3:53389 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:31.380556 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:31.375819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:57355)
0810 15:16:31.375982 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:31.375994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:31.376052 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:31.376224 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:31.376226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:31.376392 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:31.376446 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.376447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.377186 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.377187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.377732 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.377734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.378134 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.378135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.378180 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.378610 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:31.378615 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:31.380478 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":63,"threads_started":1}
W20260810 15:16:31.380666 14414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:31.375934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:57355 (local address 127.29.230.62:46589)
0810 15:16:31.376042 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:31.376044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:31.376053 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:31.376109 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:31.376110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:31.376121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:31.376169 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:31.376575 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.377098 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.377832 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.378040 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.378123 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.380601 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:57355: BlockingRecv error: recv got EOF from 127.29.230.1:57355 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:31.527889 14415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:31.522632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:52587)
0810 15:16:31.522806 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:31.522820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:31.522891 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:31.523033 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:31.523034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:31.523219 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:31.523299 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.523301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.524105 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.524106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.524641 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.524643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.525064 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.525065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.525126 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.525726 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:31.525732 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:31.527819 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
W20260810 15:16:31.528115 14414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:31.522755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:52587 (local address 127.29.230.62:46589)
0810 15:16:31.522857 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:31.522859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:31.522877 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:16:31.522942 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:31.522943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:31.522954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:31.523013 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:31.523410 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.523951 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.524741 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.524944 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.525032 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.528061 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:52587: BlockingRecv error: recv got EOF from 127.29.230.2:52587 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:31 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:31.654481 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:31.649568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:38359)
0810 15:16:31.649719 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:31.649732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:31.649796 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:31.649964 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:31.649966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:31.650136 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:31.650188 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.650190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.651082 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.651084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.651753 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:31.651757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.652148 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.652149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:31.652202 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.652690 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:31.652694 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:31.654415 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
W20260810 15:16:31.654599 14414 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:31.649710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:38359 (local address 127.29.230.62:46589)
0810 15:16:31.649800 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:31.649802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:31.649828 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:31.649855 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:31.649856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:31.649870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:31.649914 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:31.650282 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.650984 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.651832 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:31.652044 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:31.652103 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:31.654529 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:38359: BlockingRecv error: recv got EOF from 127.29.230.3:38359 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:32.389453 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:32.381075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:55203)
0810 15:16:32.381221 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:32.381235 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:32.381302 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:32.381480 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:32.381481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:32.381664 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:32.381718 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.381720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.382509 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.382510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.383096 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.383099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.383517 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.383518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.383576 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.387490 (+ 3914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:32.387495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:32.389388 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
W20260810 15:16:32.389607 14418 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:32.381159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:55203 (local address 127.29.230.62:46589)
0810 15:16:32.381277 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:32.381279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:32.381288 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:32.381374 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:32.381375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:32.381389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:32.381460 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:32.381814 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.382419 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.383192 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.383440 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.383499 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.389530 (+ 6031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:55203: BlockingRecv error: recv got EOF from 127.29.230.1:55203 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:32.533064 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:32.528403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:33019)
0810 15:16:32.528574 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:32.528587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:32.528660 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:32.528791 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:32.528792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:32.529017 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:32.529069 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.529071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.529733 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.529734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.530316 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.530319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.530729 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.530730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.530784 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.531252 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:32.531257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:32.533003 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
W20260810 15:16:32.533218 14418 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:32.528551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:33019 (local address 127.29.230.62:46589)
0810 15:16:32.528609 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:32.528611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:32.528641 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:16:32.528719 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:32.528720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:32.528731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:32.528771 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:32.529123 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.529634 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.530418 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.530639 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.530699 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.533148 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:33019: BlockingRecv error: recv got EOF from 127.29.230.2:33019 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:32 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:32.659538 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:32.654974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:52807)
0810 15:16:32.655141 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:32.655153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:32.655212 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:32.655411 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:32.655412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:32.655580 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:32.655631 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.655632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.656253 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.656254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.656785 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:32.656787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.657180 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.657181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:32.657236 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.657708 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:32.657713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:32.659478 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
W20260810 15:16:32.659634 14418 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:32.655123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:52807 (local address 127.29.230.62:46589)
0810 15:16:32.655184 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:32.655186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:32.655199 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:16:32.655323 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:32.655324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:32.655338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:32.655395 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:32.655693 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.656180 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.656874 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:32.657108 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:32.657159 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:32.659578 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:52807: BlockingRecv error: recv got EOF from 127.29.230.3:52807 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:33.394570 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:33.389927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:57919)
0810 15:16:33.390074 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:33.390086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:33.390154 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:33.390321 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:33.390322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:33.390487 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:33.390537 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.390539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.391193 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.391194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.391775 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.391777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.392165 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.392167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.392217 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.392668 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:33.392673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:33.394510 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
W20260810 15:16:33.394685 14427 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:33.390022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:57919 (local address 127.29.230.62:46589)
0810 15:16:33.390147 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:33.390149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:33.390160 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:33.390235 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:33.390235 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:33.390246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:33.390301 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:33.390592 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.391116 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.391863 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.392069 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.392136 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.394610 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:57919: BlockingRecv error: recv got EOF from 127.29.230.1:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:33.538180 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:33.533537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:37229)
0810 15:16:33.533661 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:33.533674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:33.533733 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:33.533913 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:33.533914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:33.534088 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:33.534139 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.534141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.534801 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.534802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.535441 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.535444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.535835 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.535836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.535886 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.536361 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:33.536365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:33.538120 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
W20260810 15:16:33.538275 14427 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:33.533642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:37229 (local address 127.29.230.62:46589)
0810 15:16:33.533719 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:33.533721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:33.533732 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:16:33.533795 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:33.533796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:33.533809 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:33.533859 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:33.534210 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.534724 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.535541 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.535752 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.535812 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.538216 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:37229: BlockingRecv error: recv got EOF from 127.29.230.2:37229 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:33 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:33.664655 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:33.660063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:53669)
0810 15:16:33.660195 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:33.660205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:33.660265 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:33.660411 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:33.660412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:33.660598 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:33.660648 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.660650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.661315 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.661317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.661842 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:33.661844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.662191 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.662193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:33.662249 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.662731 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:33.662736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:33.664596 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
W20260810 15:16:33.664755 14427 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:33.660167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:53669 (local address 127.29.230.62:46589)
0810 15:16:33.660220 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:33.660221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:33.660251 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:16:33.660337 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:33.660338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:33.660350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:33.660398 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:33.660727 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.661236 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.661929 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:33.662125 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:33.662181 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:33.664682 (+ 2501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:53669: BlockingRecv error: recv got EOF from 127.29.230.3:53669 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:34.399796 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:34.395048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:37855)
0810 15:16:34.395236 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:34.395249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:34.395347 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:34.395488 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:34.395489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:34.395653 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:34.395708 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.395710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.396385 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.396386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.396926 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.396927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.397295 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.397296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.397338 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.397780 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:34.397784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:34.399736 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":106,"threads_started":1}
W20260810 15:16:34.399926 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:34.395181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:37855 (local address 127.29.230.62:46589)
0810 15:16:34.395334 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:34.395336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:34.395345 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:34.395394 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:34.395395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:34.395406 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:34.395465 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:34.395776 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.396314 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.397003 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.397231 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.397287 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.399854 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:37855: BlockingRecv error: recv got EOF from 127.29.230.1:37855 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:34.543339 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:34.538680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:35885)
0810 15:16:34.538813 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:34.538827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:34.538900 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:34.539059 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:34.539060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:34.539233 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:34.539314 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.539316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.539999 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.540000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.540526 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.540527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.540871 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.540872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.540929 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.541377 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:34.541382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:34.543242 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260810 15:16:34.543428 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:34.538785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:35885 (local address 127.29.230.62:46589)
0810 15:16:34.538854 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:34.538856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:34.538880 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:16:34.538963 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:34.538963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:34.538975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:34.539027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:34.539395 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.539912 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.540609 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.540805 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.540864 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.543375 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:35885: BlockingRecv error: recv got EOF from 127.29.230.2:35885 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:34 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:34.669747 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:34.665136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:33083)
0810 15:16:34.665274 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:34.665284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:34.665341 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:34.665468 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:34.665468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:34.665642 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:34.665694 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.665695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.666373 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.666374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.666956 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:34.666958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.667389 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.667390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:34.667448 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.667856 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:34.667861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:34.669678 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260810 15:16:34.669847 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:34.665228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:33083 (local address 127.29.230.62:46589)
0810 15:16:34.665276 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:34.665278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:34.665329 (+ 51us) server_negotiation.cc:407] Connection header received
0810 15:16:34.665398 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:34.665398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:34.665411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:34.665451 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:34.665770 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.666279 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.667066 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:34.667306 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:34.667364 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:34.669785 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:33083: BlockingRecv error: recv got EOF from 127.29.230.3:33083 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:35.405028 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:35.400240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:42947)
0810 15:16:35.400396 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:35.400410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:35.400476 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:35.400628 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:35.400629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:35.400786 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:35.400838 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.400840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.401550 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.401551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.402108 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.402110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.402463 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.402464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.402515 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.403011 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:35.403016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:35.404955 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
W20260810 15:16:35.405148 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:35.400352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:42947 (local address 127.29.230.62:46589)
0810 15:16:35.400489 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:35.400491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:35.400500 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:35.400523 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:35.400524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:35.400535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:35.400600 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:35.400910 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.401450 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.402185 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.402397 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.402450 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.405065 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:42947: BlockingRecv error: recv got EOF from 127.29.230.1:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:35.548840 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:35.543902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:57925)
0810 15:16:35.544059 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:35.544070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:35.544137 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:35.544332 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:35.544334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:35.544536 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:35.544597 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.544599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.545573 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.545574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.546103 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.546104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.546459 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.546460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.546517 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.546948 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:35.546953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:35.548753 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
W20260810 15:16:35.548913 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:35.543978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:57925 (local address 127.29.230.62:46589)
0810 15:16:35.544044 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:35.544046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:35.544160 (+ 114us) server_negotiation.cc:407] Connection header received
0810 15:16:35.544196 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:35.544197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:35.544212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:35.544277 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:35.544681 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.545492 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.546181 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.546388 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.546444 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.548861 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:57925: BlockingRecv error: recv got EOF from 127.29.230.2:57925 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:35 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:35.674711 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:35.670225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:60983)
0810 15:16:35.670357 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:35.670370 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:35.670446 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:35.670607 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:35.670608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:35.670776 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:35.670846 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.670848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.671606 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.671607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.672125 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:35.672127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.672456 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.672457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:35.672511 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.672946 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:35.672950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:35.674635 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
W20260810 15:16:35.674799 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:35.670331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:60983 (local address 127.29.230.62:46589)
0810 15:16:35.670395 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:35.670397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:35.670437 (+ 40us) server_negotiation.cc:407] Connection header received
0810 15:16:35.670503 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:35.670504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:35.670519 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:35.670571 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:35.670936 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.671516 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.672202 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:35.672392 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:35.672456 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:35.674742 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:60983: BlockingRecv error: recv got EOF from 127.29.230.3:60983 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:36.409771 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:36.405474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:46639)
0810 15:16:36.405606 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:36.405617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:36.405669 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:36.405787 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:36.405787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:36.405958 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:36.406009 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.406010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.406614 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.406614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.407140 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.407141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.407508 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.407509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.407550 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.407955 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:36.407959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:36.409697 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
W20260810 15:16:36.409897 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:36.405542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:46639 (local address 127.29.230.62:46589)
0810 15:16:36.405645 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:36.405647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:36.405656 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:36.405720 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:36.405721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:36.405731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:36.405778 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:36.406061 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.406550 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.407205 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.407452 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.407502 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.409816 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:46639: BlockingRecv error: recv got EOF from 127.29.230.1:46639 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:36.553992 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:36.549343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:45671)
0810 15:16:36.549499 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:36.549509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:36.549566 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:36.549678 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:36.549679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:36.549850 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:36.549902 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.549903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.550642 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.550643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.551175 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.551176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.551561 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.551562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.551620 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.552112 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:36.552117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:36.553931 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
W20260810 15:16:36.554164 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:36.549424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:45671 (local address 127.29.230.62:46589)
0810 15:16:36.549493 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:36.549495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:36.549560 (+ 65us) server_negotiation.cc:407] Connection header received
0810 15:16:36.549610 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:36.549611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:36.549621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:36.549666 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:36.549974 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.550545 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.551281 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.551486 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.551553 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.554103 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:45671: BlockingRecv error: recv got EOF from 127.29.230.2:45671 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:36 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:36.680045 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:36.675329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:37301)
0810 15:16:36.675455 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:36.675467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:36.675524 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:36.675631 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:36.675632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:36.675792 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:36.675841 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.675842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.676524 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.676526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.677186 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:36.677189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.677579 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.677581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:36.677643 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.678098 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:36.678103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:36.679963 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
W20260810 15:16:36.680145 14448 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:36.675369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:37301 (local address 127.29.230.62:46589)
0810 15:16:36.675425 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:36.675427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:36.675521 (+ 94us) server_negotiation.cc:407] Connection header received
0810 15:16:36.675563 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:36.675564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:36.675575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:36.675626 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:36.675903 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.676443 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.677279 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:36.677482 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:36.677546 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:36.680081 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:37301: BlockingRecv error: recv got EOF from 127.29.230.3:37301 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:37.417479 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:37.410210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:47137)
0810 15:16:37.410356 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:37.410368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:37.410422 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:37.410559 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:37.410560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:37.410732 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:37.410786 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.410787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.411422 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.411423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.411953 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.411954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.412319 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.412320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.412386 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.415673 (+ 3287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:37.415679 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:37.417415 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
W20260810 15:16:37.417609 14461 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:37.410311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47137 (local address 127.29.230.62:46589)
0810 15:16:37.410417 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:37.410418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:37.410428 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:37.410468 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:37.410469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:37.410486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:37.410542 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:37.410842 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.411371 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.412023 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.412229 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.412279 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.417531 (+ 5252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:47137: BlockingRecv error: recv got EOF from 127.29.230.1:47137 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":50,"threads_started":1}
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:37.559125 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:37.554480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:40293)
0810 15:16:37.554625 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:37.554637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:37.554698 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:37.554820 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:37.554821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:37.554982 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:37.555033 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.555034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.555731 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.555732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.556260 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.556261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.556615 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.556616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.556651 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.557161 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:37.557166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:37.559063 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
W20260810 15:16:37.559233 14461 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:37.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:40293 (local address 127.29.230.62:46589)
0810 15:16:37.554638 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:37.554640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:37.554684 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:16:37.554753 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:37.554754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:37.554765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:37.554804 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:37.555126 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.555669 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.556345 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.556550 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.556614 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.559158 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:40293: BlockingRecv error: recv got EOF from 127.29.230.2:40293 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:37 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:37.685148 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:37.680520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:54863)
0810 15:16:37.680693 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:37.680704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:37.680773 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:37.680933 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:37.680934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:37.681124 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:37.681177 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.681178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.681844 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.681845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.682407 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:37.682409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.682763 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.682763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:37.682809 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.683331 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:37.683337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:37.685087 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
W20260810 15:16:37.685283 14461 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:37.680679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:54863 (local address 127.29.230.62:46589)
0810 15:16:37.680724 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:37.680726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:37.680754 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:16:37.680814 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:37.680815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:37.680827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:37.680877 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:37.681250 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.681765 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.682500 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:37.682693 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:37.682747 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:37.685213 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:54863: BlockingRecv error: recv got EOF from 127.29.230.3:54863 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:38.422492 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:38.417954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:35577)
0810 15:16:38.418127 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:38.418139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:38.418194 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:38.418334 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:38.418334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:38.418505 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:38.418559 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.418560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.419179 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.419180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.419775 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.419777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.420141 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.420141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.420178 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.420612 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:38.420617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:38.422398 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
W20260810 15:16:38.422585 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:38.418109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:35577 (local address 127.29.230.62:46589)
0810 15:16:38.418220 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:38.418221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:38.418230 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:38.418253 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:38.418253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:38.418263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:38.418322 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:38.418617 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.419121 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.419869 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.420076 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.420144 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.422502 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:35577: BlockingRecv error: recv got EOF from 127.29.230.1:35577 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:38.564157 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:38.559617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:56667)
0810 15:16:38.559762 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:38.559780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:38.559838 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:38.559977 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:38.559977 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:38.560144 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:38.560198 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.560199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.560838 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.560839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.561385 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.561387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.561727 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.561728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.561766 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.562254 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:38.562258 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:38.564054 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
W20260810 15:16:38.564221 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:38.559734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:56667 (local address 127.29.230.62:46589)
0810 15:16:38.559826 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:38.559828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:38.559837 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:38.559896 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:38.559897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:38.559908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:38.559955 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:38.560249 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.560749 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.561484 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.561676 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.561721 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.564157 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:56667: BlockingRecv error: recv got EOF from 127.29.230.2:56667 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:38 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:38.689882 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:38.685607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:44111)
0810 15:16:38.685749 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:38.685759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:38.685815 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:38.685935 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:38.685935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:38.686122 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:38.686169 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.686171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.686795 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.686796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.687402 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:38.687405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.687726 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.687727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:38.687762 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.688170 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:38.688174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:38.689813 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260810 15:16:38.689973 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:38.685695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:44111 (local address 127.29.230.62:46589)
0810 15:16:38.685744 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:38.685746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:38.685802 (+ 56us) server_negotiation.cc:407] Connection header received
0810 15:16:38.685869 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:38.685870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:38.685881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:38.685922 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:38.686223 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.686718 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.687475 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:38.687671 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:38.687724 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:38.689907 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:44111: BlockingRecv error: recv got EOF from 127.29.230.3:44111 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:39.427428 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:39.422972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:48381)
0810 15:16:39.423101 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:39.423111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:39.423166 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:39.423347 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:39.423347 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:39.423516 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:39.423569 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.423570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.424205 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.424206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.424746 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.424748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.425061 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.425062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.425106 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.425550 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:39.425554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:39.427363 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260810 15:16:39.427507 14481 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:39.423049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:48381 (local address 127.29.230.62:46589)
0810 15:16:39.423185 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:39.423187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:39.423197 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:39.423225 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:39.423226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:39.423236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:39.423339 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:39.423631 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.424146 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.424808 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.425008 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.425058 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.427445 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:48381: BlockingRecv error: recv got EOF from 127.29.230.1:48381 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:39.569339 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:39.564750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:41777)
0810 15:16:39.564874 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:39.564886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:39.564938 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:39.565062 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:39.565062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:39.565240 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:39.565287 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.565289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.565930 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.565930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.566472 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.566473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.566832 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.566832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.566868 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.567350 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:39.567355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:39.569277 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
W20260810 15:16:39.569424 14481 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:39.564840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:41777 (local address 127.29.230.62:46589)
0810 15:16:39.564927 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:39.564929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:39.564939 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:39.564988 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:39.564989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:39.565000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:39.565050 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:39.565358 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.565867 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.566562 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.566769 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.566830 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.569367 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:41777: BlockingRecv error: recv got EOF from 127.29.230.2:41777 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:39 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:39.694852 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:39.690361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:49917)
0810 15:16:39.690494 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:39.690506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:39.690576 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:39.690701 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:39.690702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:39.690880 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:39.690932 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.690934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.691681 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.691682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.692234 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:39.692236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.692569 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.692570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:39.692627 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.693068 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:39.693072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:39.694762 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260810 15:16:39.694976 14481 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:39.690462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:49917 (local address 127.29.230.62:46589)
0810 15:16:39.690510 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:39.690512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:39.690557 (+ 45us) server_negotiation.cc:407] Connection header received
0810 15:16:39.690626 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:39.690627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:39.690641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:39.690688 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:39.691013 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.691573 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.692316 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:39.692507 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:39.692564 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:39.694858 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:49917: BlockingRecv error: recv got EOF from 127.29.230.3:49917 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:40.432472 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:40.427863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:56635)
0810 15:16:40.427983 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:40.427995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:40.428044 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:40.428201 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:40.428202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:40.428380 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:40.428433 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.428434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.429219 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.429221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.429778 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.429780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.430152 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.430153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.430212 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.430653 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:40.430658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:40.432411 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
W20260810 15:16:40.432577 14487 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:40.427925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:56635 (local address 127.29.230.62:46589)
0810 15:16:40.428067 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:40.428069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:40.428078 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:40.428103 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:40.428104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:40.428114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:40.428180 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:40.428499 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.429122 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.429853 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.430081 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.430134 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.432511 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:56635: BlockingRecv error: recv got EOF from 127.29.230.1:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:40.574142 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:40.569804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:42043)
0810 15:16:40.569932 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:40.569942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:40.569996 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:40.570114 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:40.570115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:40.570278 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:40.570325 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.570327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.570931 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.570931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.571551 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.571554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.571916 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.571917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.571953 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.572372 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:40.572376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:40.574073 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
W20260810 15:16:40.574295 14487 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:40.569935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:42043 (local address 127.29.230.62:46589)
0810 15:16:40.569979 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:40.569981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:40.569989 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:16:40.570046 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:40.570047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:40.570056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:40.570101 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:40.570373 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.570877 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.571645 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.571849 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.571917 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.574219 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:42043: BlockingRecv error: recv got EOF from 127.29.230.2:42043 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:40 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:40.699942 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:40.695375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:42069)
0810 15:16:40.695512 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:40.695523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:40.695591 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:40.695734 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:40.695735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:40.695907 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:40.695988 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.695990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.696614 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.696615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.697162 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:40.697164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.697441 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.697442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:40.697478 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.697905 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:40.697910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:40.699883 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
W20260810 15:16:40.700068 14487 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:40.695392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:42069 (local address 127.29.230.62:46589)
0810 15:16:40.695443 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:40.695445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:40.695592 (+ 147us) server_negotiation.cc:407] Connection header received
0810 15:16:40.695643 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:40.695644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:40.695660 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:40.695714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:40.696044 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.696541 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.697221 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:40.697394 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:40.697436 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:40.700003 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:42069: BlockingRecv error: recv got EOF from 127.29.230.3:42069 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:41.437439 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:41.432938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:55405)
0810 15:16:41.433063 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:41.433074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:41.433133 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:41.433255 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:41.433255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:41.433425 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:41.433475 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.433476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.434149 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.434150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.434711 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.434714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.435043 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.435045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.435097 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.435553 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:41.435558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:41.437350 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
W20260810 15:16:41.437525 14491 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:41.433018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:55405 (local address 127.29.230.62:46589)
0810 15:16:41.433133 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:41.433135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:41.433162 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:16:41.433183 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:41.433184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:41.433194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:41.433242 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:41.433569 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.434084 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.434772 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.434960 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.435005 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.437462 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:55405: BlockingRecv error: recv got EOF from 127.29.230.1:55405 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:41.579037 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:41.574600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:54263)
0810 15:16:41.574738 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:41.574748 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:41.574796 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:41.574925 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:41.574926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:41.575106 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:41.575156 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.575158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.575836 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.575837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.576395 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.576398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.576759 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.576759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.576797 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.577203 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:41.577208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:41.578968 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
W20260810 15:16:41.579155 14491 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:41.574684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:54263 (local address 127.29.230.62:46589)
0810 15:16:41.574766 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:41.574768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:41.574780 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:16:41.574852 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:41.574852 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:41.574863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:41.574911 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:41.575205 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.575740 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.576485 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.576700 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.576756 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.579091 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:54263: BlockingRecv error: recv got EOF from 127.29.230.2:54263 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:41 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.3@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:41.705052 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:41.700378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.3:49147)
0810 15:16:41.700506 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:41.700516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:41.700581 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:41.700711 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:41.700712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:41.700880 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:41.700928 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.700930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.701554 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.701555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.702229 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:41.702234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.702677 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.702678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:41.702729 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.703190 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:41.703195 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:41.704992 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
W20260810 15:16:41.705143 14491 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:41.700453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:49147 (local address 127.29.230.62:46589)
0810 15:16:41.700511 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:41.700513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:41.700570 (+ 57us) server_negotiation.cc:407] Connection header received
0810 15:16:41.700644 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:41.700644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:41.700657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:41.700700 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:41.700993 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.701488 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.702325 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:41.702613 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:41.702665 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:41.705081 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.3:49147: BlockingRecv error: recv got EOF from 127.29.230.3:49147 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Aug 10 15:16:42 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:42 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.1@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:42.445477 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:42.438030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.1:52687)
0810 15:16:42.438158 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:42.438169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:42.438222 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:42.438364 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:42.438365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:42.438529 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:42.438585 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.438586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.439222 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.439223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.439805 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.439808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.440152 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.440153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.440190 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.443554 (+ 3364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:42.443560 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:42.445413 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260810 15:16:42.445626 14495 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:42.438136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:52687 (local address 127.29.230.62:46589)
0810 15:16:42.438250 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:42.438251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:42.438260 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:42.438282 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:42.438282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:42.438293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:42.438348 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:42.438637 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.439152 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.439880 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.440095 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.440149 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.445534 (+ 5385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.1:52687: BlockingRecv error: recv got EOF from 127.29.230.1:52687 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
Aug 10 15:16:42 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
Aug 10 15:16:42 dist-test-slave-gjrz krb5kdc[12718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.230.2@KRBTEST.COM for oryx/127.29.230.62@KRBTEST.COM, Server not found in Kerberos database
I20260810 15:16:42.584796 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:42.579655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46589 (local address 127.29.230.2:51987)
0810 15:16:42.579808 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:42.579821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:42.579887 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:42.580073 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:42.580082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:42.580249 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:42.580304 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.580306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.581139 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.581140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.581797 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.581800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.582213 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.582214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.582262 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.582745 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:42.582751 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:42.584721 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.230.62:46589: Server oryx/127.29.230.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
W20260810 15:16:42.584913 14495 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:42.579759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:51987 (local address 127.29.230.62:46589)
0810 15:16:42.579808 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:42.579810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:42.579865 (+ 55us) server_negotiation.cc:407] Connection header received
0810 15:16:42.579961 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:42.579962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:42.579973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:42.580021 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:42.580411 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.581034 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.581915 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.582136 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.582201 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.584850 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.230.2:51987: BlockingRecv error: recv got EOF from 127.29.230.2:51987 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:42.624593 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 13706
I20260810 15:16:42.637239 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 13948
I20260810 15:16:42.647250 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 14170
I20260810 15:16:42.655018 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 13591
2026-08-10T15:16:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17693 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T15:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:42Z Disabled control of system clock
I20260810 15:16:42.683960 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36393
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 15:16:42.766547 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:42.766750 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:42.766769 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:42.768276 14501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:42.768329 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:42.768342 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:42.768353 14501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:42.768364 14501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:42.769760 14501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:42.770009 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:42.770291 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:16:42.775216 14501 server_base.cc:1034] running on GCE node
W20260810 15:16:42.775182 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:42.775218 14509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:42.775187 14510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:42.775712 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:42.775988 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:42.777149 14501 hybrid_clock.cc:648] HybridClock initialized: now 1786375002777127 us; error 38 us; skew 500 ppm
I20260810 15:16:42.778434 14501 webserver.cc:460] Webserver started at http://127.29.230.62:43609/ using document root <none> and password file <none>
I20260810 15:16:42.778641 14501 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:42.778681 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:42.778800 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:42.779882 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "ad9bcf1acfc14078b37fe35acdefeaaa"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "7caa36fdcb973b77cd510787695148a1"
server_key_iv: "7b6933bd126c6d5e01ea36cf0e7e14c2"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.780192 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "ad9bcf1acfc14078b37fe35acdefeaaa"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "7caa36fdcb973b77cd510787695148a1"
server_key_iv: "7b6933bd126c6d5e01ea36cf0e7e14c2"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.781365 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:42.782043 14522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:42.782210 14501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:42.782285 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "ad9bcf1acfc14078b37fe35acdefeaaa"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "7caa36fdcb973b77cd510787695148a1"
server_key_iv: "7b6933bd126c6d5e01ea36cf0e7e14c2"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.782341 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:42.808653 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:42.808961 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:42.809079 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:42.812945 14501 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36393
I20260810 15:16:42.813015 14612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36393 every 8 connection(s)
I20260810 15:16:42.813334 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:42.814121 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:42.816861 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa: Bootstrap starting.
I20260810 15:16:42.817508 14614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:42.817819 14614 log.cc:824] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:42.818552 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 14501
I20260810 15:16:42.818655 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:42.818799 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa: No bootstrap required, opened a new log
I20260810 15:16:42.818857 30616 external_mini_cluster.cc:1428] Setting key 56801cd7e1bd115de77b2dad437b628b
I20260810 15:16:42.820909 14614 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } }
I20260810 15:16:42.821055 14614 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:42.821079 14614 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9bcf1acfc14078b37fe35acdefeaaa, State: Initialized, Role: FOLLOWER
I20260810 15:16:42.821193 14614 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } }
I20260810 15:16:42.821283 14614 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:42.821312 14614 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:42.821362 14614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:42.821940 14614 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } }
I20260810 15:16:42.822036 14614 leader_election.cc:302] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad9bcf1acfc14078b37fe35acdefeaaa; no voters:
I20260810 15:16:42.822207 14614 leader_election.cc:288] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:42.822239 14623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:42.822404 14623 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [term 1 LEADER]: Becoming Leader. State: Replica: ad9bcf1acfc14078b37fe35acdefeaaa, State: Running, Role: LEADER
I20260810 15:16:42.822476 14614 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:42.822544 14623 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } }
I20260810 15:16:42.822908 14623 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9bcf1acfc14078b37fe35acdefeaaa. Latest consensus state: current_term: 1 leader_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } } }
I20260810 15:16:42.823117 14623 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:42.823236 14629 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9bcf1acfc14078b37fe35acdefeaaa" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36393 } } }
I20260810 15:16:42.823658 14629 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:42.823844 14646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:42.824365 14646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:42.825474 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:42.819455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.29.230.62:36393)
0810 15:16:42.819675 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:42.819682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:42.819708 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:16:42.820165 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:42.820173 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:42.820233 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:42.820358 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:42.820584 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.822258 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.823326 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.823925 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.824050 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.824776 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:42.824780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:42.824780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:42.824953 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:42.824988 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:42.824999 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:42.825060 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:42.825181 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 15:16:42.825260 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260810 15:16:42.826020 14646 catalog_manager.cc:1347] Generated new cluster ID: 92f9e78be43b4b15a9bae4af4681ac59
I20260810 15:16:42.826113 14646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:42.851150 14646 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:42.851966 14646 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:42.857347 14646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad9bcf1acfc14078b37fe35acdefeaaa: Generated new TSK 0
I20260810 15:16:42.857594 14646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:42.864519 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--builtin_ntp_servers=127.29.230.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 15:16:42.947120 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:42.947363 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:42.947402 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:42.948578 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:42.948621 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:42.948654 14659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:42.950007 14659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:42.950253 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:42.950557 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:42.955536 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:42.955521 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:42.955571 14659 server_base.cc:1034] running on GCE node
W20260810 15:16:42.955505 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:42.955899 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:42.956156 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:42.957285 14659 hybrid_clock.cc:648] HybridClock initialized: now 1786375002957271 us; error 30 us; skew 500 ppm
I20260810 15:16:42.958477 14659 webserver.cc:460] Webserver started at http://127.29.230.1:35409/ using document root <none> and password file <none>
I20260810 15:16:42.958701 14659 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:42.958747 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:42.958873 14659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:42.959959 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "bf47c2f9545b44cda4233daaba92948e"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "8fad3677b2cde79f7dae1af3052b604b"
server_key_iv: "accfba095030150f5a89e73d1b2a3190"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.960292 14659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "bf47c2f9545b44cda4233daaba92948e"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "8fad3677b2cde79f7dae1af3052b604b"
server_key_iv: "accfba095030150f5a89e73d1b2a3190"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.961510 14659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:42.962214 14682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:42.962356 14659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:42.962440 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "bf47c2f9545b44cda4233daaba92948e"
format_stamp: "Formatted at 2026-08-10 15:16:42 on dist-test-slave-gjrz"
server_key: "8fad3677b2cde79f7dae1af3052b604b"
server_key_iv: "accfba095030150f5a89e73d1b2a3190"
server_key_version: "encryptionkey@0"
I20260810 15:16:42.962502 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:42.973258 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:42.973582 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:42.973708 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:42.973902 14659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:42.974212 14659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:42.974246 14659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:42.974273 14659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:42.974298 14659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:42.979945 14659 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:41801
I20260810 15:16:42.979997 14871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:41801 every 8 connection(s)
I20260810 15:16:42.980341 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:42.985476 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:42.981046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:39361 (local address 127.29.230.62:36393)
0810 15:16:42.981092 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:42.981093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:42.981341 (+ 248us) server_negotiation.cc:407] Connection header received
0810 15:16:42.981862 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:42.981864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:42.981882 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:42.981947 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:42.982618 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.983139 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.984138 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.984335 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.984392 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.985209 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:42.985210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:42.985211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:42.985219 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:42.985236 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:42.985240 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:42.985282 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:42.985357 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:16:42.985394 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:16:42.985684 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:42.981038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36393 (local address 127.29.230.1:39361)
0810 15:16:42.981221 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:42.981254 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:42.981755 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:42.982024 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:42.982028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:42.982037 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:42.982530 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.982535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.983224 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.983226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.984052 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:42.984055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:42.984430 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:42.984431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:42.984559 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:42.985053 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:42.985054 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:42.985083 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:42.985106 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:42.985289 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:42.985295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:42.985298 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:42.985356 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:16:42.985461 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260810 15:16:42.986171 14872 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36393
I20260810 15:16:42.986253 14872 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:42.986423 14872 heartbeater.cc:511] Master 127.29.230.62:36393 requested a full tablet report, sending...
I20260810 15:16:42.986910 14546 ts_manager.cc:194] Registered new tserver with Master: bf47c2f9545b44cda4233daaba92948e (127.29.230.1:41801)
I20260810 15:16:42.987567 14546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.1:39361
I20260810 15:16:42.990381 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 14659
I20260810 15:16:42.990468 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:42.990664 30616 external_mini_cluster.cc:1428] Setting key a5871c5d98e7cdb5578430d92f014a61
I20260810 15:16:42.991537 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--builtin_ntp_servers=127.29.230.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 15:16:43.075114 14876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.075356 14876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.075379 14876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.076498 14876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.076530 14876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.076553 14876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:43.077852 14876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.078068 14876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.078342 14876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.083319 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.083325 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.083339 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.083384 14876 server_base.cc:1034] running on GCE node
I20260810 15:16:43.083726 14876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.083966 14876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.085105 14876 hybrid_clock.cc:648] HybridClock initialized: now 1786375003085087 us; error 36 us; skew 500 ppm
I20260810 15:16:43.086333 14876 webserver.cc:460] Webserver started at http://127.29.230.2:40401/ using document root <none> and password file <none>
I20260810 15:16:43.086550 14876 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.086597 14876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.086750 14876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.087913 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "5390a68c0c514fe88d6109b44139c892"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d52273be94bb6f8688b9c4cd8e1afe1d"
server_key_iv: "e966992efc37024dd32926a214b66793"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.088348 14876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "5390a68c0c514fe88d6109b44139c892"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d52273be94bb6f8688b9c4cd8e1afe1d"
server_key_iv: "e966992efc37024dd32926a214b66793"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.089702 14876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:43.090435 14900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.090651 14876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:43.090744 14876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "5390a68c0c514fe88d6109b44139c892"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d52273be94bb6f8688b9c4cd8e1afe1d"
server_key_iv: "e966992efc37024dd32926a214b66793"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.090816 14876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.109042 14876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.109339 14876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.109470 14876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.109694 14876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:43.110025 14876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:43.110060 14876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.110090 14876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:43.110107 14876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.115935 14876 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:45545
I20260810 15:16:43.115980 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:45545 every 8 connection(s)
I20260810 15:16:43.116319 14876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:43.116897 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 14876
I20260810 15:16:43.116986 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:43.117211 30616 external_mini_cluster.cc:1428] Setting key ff085994be9145aca293eee7a430d437
I20260810 15:16:43.118218 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--builtin_ntp_servers=127.29.230.20:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 15:16:43.122059 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.117049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:52795 (local address 127.29.230.62:36393)
0810 15:16:43.117109 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.117111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.117408 (+ 297us) server_negotiation.cc:407] Connection header received
0810 15:16:43.118006 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.118007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.118018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.118055 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.118950 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.119509 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.120658 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.120868 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.120932 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:43.121773 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.121775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.121775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.121781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.121799 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.121804 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.121859 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.121935 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.121967 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:16:43.122242 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.117041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36393 (local address 127.29.230.2:52795)
0810 15:16:43.117262 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:43.117309 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:43.117906 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:43.118133 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:43.118137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:43.118148 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:43.118843 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:43.118851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.119598 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.119599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:43.120544 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:43.120548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.120972 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.120973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:43.121120 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:43.121631 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:43.121631 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:43.121661 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:43.121685 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:43.121864 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:43.121867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:43.121869 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:43.121926 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:16:43.122041 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260810 15:16:43.122666 15115 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36393
I20260810 15:16:43.122788 15115 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:43.122969 15115 heartbeater.cc:511] Master 127.29.230.62:36393 requested a full tablet report, sending...
I20260810 15:16:43.123356 14546 ts_manager.cc:194] Registered new tserver with Master: 5390a68c0c514fe88d6109b44139c892 (127.29.230.2:45545)
I20260810 15:16:43.123715 14546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.2:52795
W20260810 15:16:43.201376 15120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.201601 15120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.201632 15120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.202796 15120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.202838 15120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.202869 15120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:43.204280 15120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.204525 15120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.204807 15120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.209811 15138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.209811 15131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.209996 15136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.210848 15120 server_base.cc:1034] running on GCE node
I20260810 15:16:43.211100 15120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.211364 15120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.212538 15120 hybrid_clock.cc:648] HybridClock initialized: now 1786375003212522 us; error 28 us; skew 500 ppm
I20260810 15:16:43.213910 15120 webserver.cc:460] Webserver started at http://127.29.230.3:38097/ using document root <none> and password file <none>
I20260810 15:16:43.214148 15120 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.214200 15120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.214329 15120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.215533 15120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "2347ea7c82254375861d30f1703c62d3"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d16491620f85a187df5bbb6bcc6c0411"
server_key_iv: "51180b87e2f93e802bfeecb0494840e0"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.215881 15120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "2347ea7c82254375861d30f1703c62d3"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d16491620f85a187df5bbb6bcc6c0411"
server_key_iv: "51180b87e2f93e802bfeecb0494840e0"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.217196 15120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:43.217926 15145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.218155 15120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:43.218250 15120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "2347ea7c82254375861d30f1703c62d3"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "d16491620f85a187df5bbb6bcc6c0411"
server_key_iv: "51180b87e2f93e802bfeecb0494840e0"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.218314 15120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.228315 15120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.228606 15120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.228730 15120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.228950 15120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:43.229265 15120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:43.229300 15120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.229328 15120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:43.229352 15120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.235025 15120 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:35371
I20260810 15:16:43.235087 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:35371 every 8 connection(s)
I20260810 15:16:43.235481 15120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:43.240581 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.236229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:42495 (local address 127.29.230.62:36393)
0810 15:16:43.236286 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.236289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.236489 (+ 200us) server_negotiation.cc:407] Connection header received
0810 15:16:43.237021 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.237022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.237037 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.237105 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.237773 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.238281 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.239289 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.239484 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.239541 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:43.240317 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.240318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.240319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.240324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.240340 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.240345 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.240390 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.240470 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.240499 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 15:16:43.240808 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.236200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36393 (local address 127.29.230.3:42495)
0810 15:16:43.236370 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:43.236404 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:43.236916 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:43.237177 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:43.237180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:43.237188 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:43.237677 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:43.237681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.238368 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.238368 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:43.239169 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:43.239172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:43.239555 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:43.239556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:43.239696 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:43.240176 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:43.240177 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:43.240206 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:43.240230 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:43.240397 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:43.240402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:43.240405 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:43.240466 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 15:16:43.240572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260810 15:16:43.241261 15333 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36393
I20260810 15:16:43.241348 15333 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:43.241529 15333 heartbeater.cc:511] Master 127.29.230.62:36393 requested a full tablet report, sending...
I20260810 15:16:43.241963 14546 ts_manager.cc:194] Registered new tserver with Master: 2347ea7c82254375861d30f1703c62d3 (127.29.230.3:35371)
I20260810 15:16:43.242337 14546 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.3:42495
I20260810 15:16:43.244325 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 15120
I20260810 15:16:43.244403 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:43.244587 30616 external_mini_cluster.cc:1428] Setting key fb4ebb4825af8badf5719141e6462e3b
I20260810 15:16:43.245489 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:16:43.247241 14622 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:43.246823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.29.230.62:36393)
0810 15:16:43.246922 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.246923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.246933 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:43.246994 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.246995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.247004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.247049 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.247176 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51314: BlockingRecv error: recv got EOF from 127.0.0.1:51314 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 15:16:43.247296 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 14659
I20260810 15:16:43.261093 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 14876
I20260810 15:16:43.272907 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 15120
I20260810 15:16:43.283982 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 14501
2026-08-10T15:16:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (632 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T15:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:43Z Disabled control of system clock
I20260810 15:16:43.316771 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35147
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:16:43.399657 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.399857 15350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.399878 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.401041 15350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:43.401082 15350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.401093 15350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.401104 15350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:43.401115 15350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:43.402482 15350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:35147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.402729 15350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.403003 15350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.407799 15358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.407791 15357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.407956 15350 server_base.cc:1034] running on GCE node
W20260810 15:16:43.407817 15360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.408308 15350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.408578 15350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.409724 15350 hybrid_clock.cc:648] HybridClock initialized: now 1786375003409711 us; error 30 us; skew 500 ppm
I20260810 15:16:43.411145 15350 webserver.cc:460] Webserver started at http://127.29.230.62:44545/ using document root <none> and password file <none>
I20260810 15:16:43.411371 15350 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.411417 15350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.411507 15350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.412529 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "13eb99c5d8d640289761fd3d15b3be8c"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "7ff751aafd206a3a20c40605ac20fd71"
server_key_iv: "5694f75cdfd3a12088187b4d8c453863"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.412832 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "13eb99c5d8d640289761fd3d15b3be8c"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "7ff751aafd206a3a20c40605ac20fd71"
server_key_iv: "5694f75cdfd3a12088187b4d8c453863"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.413980 15350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:16:43.414673 15371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.414821 15350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:16:43.414908 15350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "13eb99c5d8d640289761fd3d15b3be8c"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "7ff751aafd206a3a20c40605ac20fd71"
server_key_iv: "5694f75cdfd3a12088187b4d8c453863"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.414996 15350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.418222 15350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.418421 15350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.418512 15350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.422065 15350 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:35147
I20260810 15:16:43.422129 15457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:35147 every 8 connection(s)
I20260810 15:16:43.422400 15350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:43.423043 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:43.425767 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c: Bootstrap starting.
I20260810 15:16:43.426343 15459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:43.426625 15459 log.cc:824] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:43.427320 15459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c: No bootstrap required, opened a new log
I20260810 15:16:43.428906 15459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } }
I20260810 15:16:43.429036 15459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:43.429060 15459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13eb99c5d8d640289761fd3d15b3be8c, State: Initialized, Role: FOLLOWER
I20260810 15:16:43.429200 15459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } }
I20260810 15:16:43.429267 15459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:43.429287 15459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:43.429332 15459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:43.429838 15459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } }
I20260810 15:16:43.429924 15459 leader_election.cc:302] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13eb99c5d8d640289761fd3d15b3be8c; no voters:
I20260810 15:16:43.430073 15459 leader_election.cc:288] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:43.430115 15465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:43.430266 15465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [term 1 LEADER]: Becoming Leader. State: Replica: 13eb99c5d8d640289761fd3d15b3be8c, State: Running, Role: LEADER
I20260810 15:16:43.430322 15459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:43.430358 15465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } }
I20260810 15:16:43.430730 15466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13eb99c5d8d640289761fd3d15b3be8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } } }
I20260810 15:16:43.430817 15466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:43.430945 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13eb99c5d8d640289761fd3d15b3be8c. Latest consensus state: current_term: 1 leader_uuid: "13eb99c5d8d640289761fd3d15b3be8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13eb99c5d8d640289761fd3d15b3be8c" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35147 } } }
I20260810 15:16:43.431047 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:43.431142 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 15350
I20260810 15:16:43.431190 15481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:43.431241 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:43.431454 30616 external_mini_cluster.cc:1428] Setting key 55dd7b80d70a40100aee2c2f860ad75b
I20260810 15:16:43.431727 15481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:43.433642 15481 catalog_manager.cc:1347] Generated new cluster ID: 2d3f47ba36ee4fcfba3c2be74084cb0b
I20260810 15:16:43.433698 15481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:43.435828 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.432122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60056 (local address 127.29.230.62:35147)
0810 15:16:43.432323 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.432333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.432367 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:16:43.432825 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.432833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.432875 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.433011 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.435356 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.435359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.435360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.435364 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.435405 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.435415 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.435486 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.435582 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.435647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260810 15:16:43.442076 15481 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:43.442226 15481 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:43.446081 15481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13eb99c5d8d640289761fd3d15b3be8c: Generated new TSK 0
I20260810 15:16:43.446278 15481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:16:43.450878 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--builtin_ntp_servers=127.29.230.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:16:43.536305 15500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.536512 15500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.536545 15500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.537703 15500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.537743 15500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.537777 15500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:43.539127 15500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.539383 15500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.539671 15500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.544626 15506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.544636 15511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.544715 15507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.545073 15500 server_base.cc:1034] running on GCE node
I20260810 15:16:43.545284 15500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.545531 15500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.546677 15500 hybrid_clock.cc:648] HybridClock initialized: now 1786375003546661 us; error 30 us; skew 500 ppm
I20260810 15:16:43.547906 15500 webserver.cc:460] Webserver started at http://127.29.230.1:43183/ using document root <none> and password file <none>
I20260810 15:16:43.548138 15500 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.548190 15500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.548321 15500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.549403 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "6f896f70524848dd9c62721bda8d12d0"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "a624e1b839457c2ba80f8d655dd75065"
server_key_iv: "ed5edbd07bb86e89fd6ef6fe7a87eb8a"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.549759 15500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "6f896f70524848dd9c62721bda8d12d0"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "a624e1b839457c2ba80f8d655dd75065"
server_key_iv: "ed5edbd07bb86e89fd6ef6fe7a87eb8a"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.551074 15500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:16:43.551887 15525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.552049 15500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.552142 15500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "6f896f70524848dd9c62721bda8d12d0"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "a624e1b839457c2ba80f8d655dd75065"
server_key_iv: "ed5edbd07bb86e89fd6ef6fe7a87eb8a"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.552214 15500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.555466 15500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.555730 15500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.555853 15500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.556046 15500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:43.556337 15500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:43.556375 15500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.556401 15500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:43.556427 15500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.561853 15500 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:42957
I20260810 15:16:43.561930 15722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:42957 every 8 connection(s)
I20260810 15:16:43.562237 15500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:43.564956 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.562909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:44393 (local address 127.29.230.62:35147)
0810 15:16:43.562950 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.562952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.563201 (+ 249us) server_negotiation.cc:407] Connection header received
0810 15:16:43.563708 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.563709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.563716 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.563751 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.564778 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.564779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.564779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.564783 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.564797 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.564801 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.564844 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.564890 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.564902 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:16:43.565164 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.562894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35147 (local address 127.29.230.1:44393)
0810 15:16:43.563085 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:43.563117 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:43.563609 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:43.563813 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:43.563816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:43.563824 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:43.564644 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:43.564645 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:43.564674 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:43.564702 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:43.564844 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:43.564846 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:43.564848 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:43.564896 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:16:43.564943 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260810 15:16:43.565629 15724 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35147
I20260810 15:16:43.565703 15724 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:43.565841 15724 heartbeater.cc:511] Master 127.29.230.62:35147 requested a full tablet report, sending...
I20260810 15:16:43.566310 15401 ts_manager.cc:194] Registered new tserver with Master: 6f896f70524848dd9c62721bda8d12d0 (127.29.230.1:42957)
I20260810 15:16:43.566428 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 15500
I20260810 15:16:43.566514 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:43.566692 30616 external_mini_cluster.cc:1428] Setting key 8c0ecb92136f56018225a74f77fd7a4f
I20260810 15:16:43.567516 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--builtin_ntp_servers=127.29.230.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:16:43.648643 15727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.648859 15727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.648890 15727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.650040 15727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.650086 15727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.650121 15727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:43.651513 15727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.651773 15727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.652077 15727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.657150 15735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.657160 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.657177 15738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.657330 15727 server_base.cc:1034] running on GCE node
I20260810 15:16:43.657541 15727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.657774 15727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.658916 15727 hybrid_clock.cc:648] HybridClock initialized: now 1786375003658899 us; error 28 us; skew 500 ppm
I20260810 15:16:43.660228 15727 webserver.cc:460] Webserver started at http://127.29.230.2:37833/ using document root <none> and password file <none>
I20260810 15:16:43.660457 15727 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.660511 15727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.660647 15727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.661703 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "ed467b188da94696b965d66e0b8227a6"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "bb695c53b13a936c6218e19e505a8dff"
server_key_iv: "abd91c95c0da54f73380dcbf6236eec5"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.662057 15727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "ed467b188da94696b965d66e0b8227a6"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "bb695c53b13a936c6218e19e505a8dff"
server_key_iv: "abd91c95c0da54f73380dcbf6236eec5"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.663465 15727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:43.664279 15747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.664453 15727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:43.664539 15727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "ed467b188da94696b965d66e0b8227a6"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "bb695c53b13a936c6218e19e505a8dff"
server_key_iv: "abd91c95c0da54f73380dcbf6236eec5"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.664608 15727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.668001 15727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.668249 15727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.668372 15727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.668565 15727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:43.668864 15727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:43.668905 15727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.668931 15727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:43.668957 15727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.674480 15727 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:33413
I20260810 15:16:43.674554 15915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:33413 every 8 connection(s)
I20260810 15:16:43.674834 15727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:16:43.677750 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.675579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:32915 (local address 127.29.230.62:35147)
0810 15:16:43.675638 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.675640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.675838 (+ 198us) server_negotiation.cc:407] Connection header received
0810 15:16:43.676317 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.676319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.676326 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.676378 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.677502 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.677504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.677504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.677509 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.677530 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.677535 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.677586 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.677663 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.677678 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:16:43.677935 15921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.675556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35147 (local address 127.29.230.2:32915)
0810 15:16:43.675722 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:43.675754 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:43.676212 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:43.676447 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:43.676451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:43.676459 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:43.677348 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:43.677348 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:43.677378 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:43.677407 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:43.677592 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:43.677598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:43.677600 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:43.677668 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:16:43.677729 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260810 15:16:43.678390 15918 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35147
I20260810 15:16:43.678468 15918 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:43.678584 15918 heartbeater.cc:511] Master 127.29.230.62:35147 requested a full tablet report, sending...
I20260810 15:16:43.678901 15401 ts_manager.cc:194] Registered new tserver with Master: ed467b188da94696b965d66e0b8227a6 (127.29.230.2:33413)
I20260810 15:16:43.682794 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 15727
I20260810 15:16:43.682880 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:43.683099 30616 external_mini_cluster.cc:1428] Setting key 914376799b10b9464832cbb47a70a7d5
I20260810 15:16:43.684031 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--builtin_ntp_servers=127.29.230.20:35347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 15:16:43.766232 15922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:43.766467 15922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:43.766499 15922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:43.767771 15922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:43.767812 15922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:43.767848 15922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:43.769268 15922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:43.769515 15922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:43.769824 15922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:43.774991 15933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:43.774984 15931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.775084 15922 server_base.cc:1034] running on GCE node
W20260810 15:16:43.774966 15936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:43.775359 15922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:43.775609 15922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:43.776767 15922 hybrid_clock.cc:648] HybridClock initialized: now 1786375003776749 us; error 33 us; skew 500 ppm
I20260810 15:16:43.778002 15922 webserver.cc:460] Webserver started at http://127.29.230.3:42441/ using document root <none> and password file <none>
I20260810 15:16:43.778280 15922 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:43.778327 15922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:43.778457 15922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:43.779580 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "5b7d5061e2464977a35253e954a11e1d"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "4eded7048bb901a40a3a03a334c364f3"
server_key_iv: "832d1bb3b983adecb1ce51f104d241ff"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.779917 15922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "5b7d5061e2464977a35253e954a11e1d"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "4eded7048bb901a40a3a03a334c364f3"
server_key_iv: "832d1bb3b983adecb1ce51f104d241ff"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.781188 15922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:43.781967 15947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.782143 15922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:43.782231 15922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "5b7d5061e2464977a35253e954a11e1d"
format_stamp: "Formatted at 2026-08-10 15:16:43 on dist-test-slave-gjrz"
server_key: "4eded7048bb901a40a3a03a334c364f3"
server_key_iv: "832d1bb3b983adecb1ce51f104d241ff"
server_key_version: "encryptionkey@0"
I20260810 15:16:43.782302 15922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:43.785708 15922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:43.785954 15922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:43.786087 15922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:43.786274 15922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:43.786572 15922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:43.786608 15922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.786634 15922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:43.786651 15922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:43.792243 15922 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:37471
I20260810 15:16:43.792303 16126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:37471 every 8 connection(s)
I20260810 15:16:43.792605 15922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:43.795420 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.793308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:43091 (local address 127.29.230.62:35147)
0810 15:16:43.793365 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.793366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.793580 (+ 214us) server_negotiation.cc:407] Connection header received
0810 15:16:43.794024 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.794025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.794032 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.794088 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.795179 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:43.795181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:43.795181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:16:43.795186 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:43.795201 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:16:43.795205 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:43.795253 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:43.795351 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 15:16:43.795364 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:16:43.795578 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:43.793292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35147 (local address 127.29.230.3:43091)
0810 15:16:43.793460 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:43.793494 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:43.793938 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:43.794156 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:43.794159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:43.794168 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:43.795001 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:16:43.795001 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:43.795031 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:16:43.795090 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:43.795252 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:43.795254 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:43.795257 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:16:43.795347 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 15:16:43.795397 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260810 15:16:43.796036 16128 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35147
I20260810 15:16:43.796125 16128 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:43.796272 16128 heartbeater.cc:511] Master 127.29.230.62:35147 requested a full tablet report, sending...
I20260810 15:16:43.796619 15401 ts_manager.cc:194] Registered new tserver with Master: 5b7d5061e2464977a35253e954a11e1d (127.29.230.3:37471)
I20260810 15:16:43.799407 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 15922
I20260810 15:16:43.799489 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:43.799683 30616 external_mini_cluster.cc:1428] Setting key 64f4fd2ea1932b8e201029891ee94ed9
I20260810 15:16:43.800510 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 15:16:43.802318 15495 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:16:43.801779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.29.230.62:35147)
0810 15:16:43.801827 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:43.801829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:43.801954 (+ 125us) server_negotiation.cc:407] Connection header received
0810 15:16:43.802059 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:43.802060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:43.802066 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:43.802118 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:43.802262 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60058: BlockingRecv error: recv got EOF from 127.0.0.1:60058 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:16:43.802407 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 15500
I20260810 15:16:43.824591 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 15727
I20260810 15:16:43.836021 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 15922
I20260810 15:16:43.848912 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 15350
2026-08-10T15:16:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (567 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:43 dist-test-slave-gjrz krb5kdc[16143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:43 dist-test-slave-gjrz krb5kdc[16143](info): set up 2 sockets
Aug 10 15:16:43 dist-test-slave-gjrz krb5kdc[16143](info): commencing operation
krb5kdc: starting...
W20260810 15:16:45.044672 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:16:45.166246 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:38923
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:45.248872 16161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:45.249082 16161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:45.249114 16161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:45.250313 16161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:16:45.250370 16161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:45.250389 16161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:45.250406 16161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:16:45.250422 16161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:16:45.252602 16161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:45.252888 16161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:45.253175 16161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:45.261945 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.261909 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.262189 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:45.262205 16161 server_base.cc:1034] running on GCE node
I20260810 15:16:45.262483 16161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:45.262807 16161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:45.263914 16161 hybrid_clock.cc:648] HybridClock initialized: now 1786375005263896 us; error 35 us; skew 500 ppm
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:45.266767 16161 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:16:45.267220 16161 webserver.cc:460] Webserver started at http://127.29.230.62:32791/ using document root <none> and password file <none>
I20260810 15:16:45.267534 16161 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:45.267598 16161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:45.267724 16161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:45.269093 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "f75491e79b654a079f05260b07c783b2"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "284a43bc29a40a39d5ef53f8e4cb7a2a"
server_key_iv: "be3ea58084e13ea565fba779e39a1a19"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.269452 16161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "f75491e79b654a079f05260b07c783b2"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "284a43bc29a40a39d5ef53f8e4cb7a2a"
server_key_iv: "be3ea58084e13ea565fba779e39a1a19"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.270704 16161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:16:45.271517 16188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.271696 16161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:16:45.271783 16161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "f75491e79b654a079f05260b07c783b2"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "284a43bc29a40a39d5ef53f8e4cb7a2a"
server_key_iv: "be3ea58084e13ea565fba779e39a1a19"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.271852 16161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:45.289330 16161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:45.291970 16161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:45.292133 16161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:45.296895 16161 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:38923
I20260810 15:16:45.296944 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:38923 every 8 connection(s)
I20260810 15:16:45.297348 16161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:16:45.298636 16276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:45.301367 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 16161
I20260810 15:16:45.301455 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:16:45.301640 30616 external_mini_cluster.cc:1428] Setting key 02606996038e2013ffc579d2cee15000
I20260810 15:16:45.302129 16276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: Bootstrap starting.
I20260810 15:16:45.302898 16276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:45.303232 16276 log.cc:824] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:45.304032 16276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: No bootstrap required, opened a new log
I20260810 15:16:45.306650 16276 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } }
I20260810 15:16:45.306831 16276 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:45.306877 16276 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75491e79b654a079f05260b07c783b2, State: Initialized, Role: FOLLOWER
I20260810 15:16:45.307034 16276 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } }
I20260810 15:16:45.307135 16276 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:16:45.307178 16276 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:16:45.307222 16276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:45.307885 16276 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } }
I20260810 15:16:45.307978 16276 leader_election.cc:302] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75491e79b654a079f05260b07c783b2; no voters:
I20260810 15:16:45.308116 16276 leader_election.cc:288] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:16:45.308210 16282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:45.308346 16276 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:16:45.308471 16282 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [term 1 LEADER]: Becoming Leader. State: Replica: f75491e79b654a079f05260b07c783b2, State: Running, Role: LEADER
I20260810 15:16:45.308624 16282 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } }
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:45.309096 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75491e79b654a079f05260b07c783b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } } }
I20260810 15:16:45.309170 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:16:45.309320 16285 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75491e79b654a079f05260b07c783b2. Latest consensus state: current_term: 1 leader_uuid: "f75491e79b654a079f05260b07c783b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75491e79b654a079f05260b07c783b2" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 38923 } } }
I20260810 15:16:45.309387 16285 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2 [sys.catalog]: This master's current role is: LEADER
W20260810 15:16:45.310073 16304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:16:45.310212 16304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:16:45.310298 16293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:16:45.310463 16293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:16:45.312115 16293 catalog_manager.cc:1347] Generated new cluster ID: 0220ce645cf4449ea7490eb27b2f6f1f
I20260810 15:16:45.312175 16293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:16:45.313386 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.302225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.29.230.62:38923)
0810 15:16:45.302409 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.302420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.302452 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:16:45.302995 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.303005 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.303079 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.303221 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:45.303908 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.305338 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.306119 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.306709 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.306807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.309746 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:45.309750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:45.309751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:45.309764 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:45.311883 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.312292 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.312296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.312296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.312382 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.312575 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.312575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.312576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.312813 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:45.312968 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.313084 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.313220 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260810 15:16:45.327956 16293 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:16:45.328490 16293 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:16:45.341986 16293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: Generated new TSK 0
I20260810 15:16:45.342195 16293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:16:45.375891 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:45.457213 16310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:45.457407 16310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:45.457437 16310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:45.458572 16310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:45.458617 16310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:45.458652 16310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:16:45.460106 16310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:45.460341 16310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:45.460558 16310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:45.465461 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.465421 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.465421 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:45.465541 16310 server_base.cc:1034] running on GCE node
I20260810 15:16:45.465889 16310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:45.466125 16310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:45.467299 16310 hybrid_clock.cc:648] HybridClock initialized: now 1786375005467281 us; error 29 us; skew 500 ppm
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:45.469816 16310 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:16:45.470256 16310 webserver.cc:460] Webserver started at http://127.29.230.1:36431/ using document root <none> and password file <none>
I20260810 15:16:45.470475 16310 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:45.470525 16310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:45.470651 16310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:45.471767 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "7d41f047f76e4317b13ac0fc0b91eff6"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "d22f7ecc48f3c6a1cff44f8f2e7956f1"
server_key_iv: "cf702e9e1e1a7cb31efdfb562a818cbc"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.472108 16310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "7d41f047f76e4317b13ac0fc0b91eff6"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "d22f7ecc48f3c6a1cff44f8f2e7956f1"
server_key_iv: "cf702e9e1e1a7cb31efdfb562a818cbc"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.473381 16310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:16:45.474104 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.474264 16310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:16:45.474352 16310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "7d41f047f76e4317b13ac0fc0b91eff6"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "d22f7ecc48f3c6a1cff44f8f2e7956f1"
server_key_iv: "cf702e9e1e1a7cb31efdfb562a818cbc"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.474426 16310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:45.500386 16310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:45.502419 16310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:45.502555 16310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:45.502782 16310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:45.503104 16310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:45.503139 16310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.503167 16310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:45.503191 16310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.508937 16310 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44739
I20260810 15:16:45.509006 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44739 every 8 connection(s)
I20260810 15:16:45.509394 16310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:16:45.511694 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 16310
I20260810 15:16:45.511781 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:16:45.511971 30616 external_mini_cluster.cc:1428] Setting key f80554e662d9ec8be5de65a504537cdb
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:45.519353 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.510186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:43639 (local address 127.29.230.62:38923)
0810 15:16:45.510264 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.510266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.510428 (+ 162us) server_negotiation.cc:407] Connection header received
0810 15:16:45.510964 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.510966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.510977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.511042 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:45.512104 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.512792 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.513812 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.514053 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.514125 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.516542 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:45.516544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:45.516544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:45.516551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:45.517868 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.518348 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.518349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.518349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.518395 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.518670 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.518670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.518670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.518773 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:45.518822 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.519152 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.519227 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:16:45.519642 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.510162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38923 (local address 127.29.230.1:43639)
0810 15:16:45.510309 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.510347 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.510855 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.511110 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.511114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.511776 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:45.512012 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.512019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.512885 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.512886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.513730 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.513733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.514173 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.514175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.514340 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.514834 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:45.514843 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:45.516449 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:45.517952 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.517961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.517964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.518269 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.518456 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.518457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.518457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.518610 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.518831 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:45.518837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:45.519006 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.519123 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.519331 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260810 15:16:45.520184 16530 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38923
I20260810 15:16:45.520273 16530 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:45.520447 16530 heartbeater.cc:511] Master 127.29.230.62:38923 requested a full tablet report, sending...
I20260810 15:16:45.520982 16218 ts_manager.cc:194] Registered new tserver with Master: 7d41f047f76e4317b13ac0fc0b91eff6 (127.29.230.1:44739)
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:16:45.521632 16218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:43639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:16:45.538581 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:45.621177 16539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:45.621380 16539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:45.621402 16539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:45.622541 16539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:45.622583 16539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:45.622609 16539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:16:45.624126 16539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:45.624343 16539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:45.624559 16539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:45.629276 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.629310 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.629276 16549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:45.629601 16539 server_base.cc:1034] running on GCE node
I20260810 15:16:45.629774 16539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:45.629966 16539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:45.631145 16539 hybrid_clock.cc:648] HybridClock initialized: now 1786375005631128 us; error 29 us; skew 500 ppm
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:45.633692 16539 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:16:45.634097 16539 webserver.cc:460] Webserver started at http://127.29.230.2:41349/ using document root <none> and password file <none>
I20260810 15:16:45.634289 16539 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:45.634336 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:45.634450 16539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:45.635494 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "878cdc49d1f2445b8bb08fc9b01732e9"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "f3de0e49430e39f2353f114a79cd21fd"
server_key_iv: "14b5c130e20bc3bb70048e6eba94de3a"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.635825 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "878cdc49d1f2445b8bb08fc9b01732e9"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "f3de0e49430e39f2353f114a79cd21fd"
server_key_iv: "14b5c130e20bc3bb70048e6eba94de3a"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.637135 16539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:45.637907 16566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.638085 16539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:16:45.638170 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "878cdc49d1f2445b8bb08fc9b01732e9"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "f3de0e49430e39f2353f114a79cd21fd"
server_key_iv: "14b5c130e20bc3bb70048e6eba94de3a"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.638229 16539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:45.655519 16539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:45.657506 16539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:45.657639 16539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:45.657851 16539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:45.658145 16539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:45.658179 16539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.658206 16539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:45.658229 16539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.663921 16539 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:46721
I20260810 15:16:45.663997 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:46721 every 8 connection(s)
I20260810 15:16:45.664283 16539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:45.672930 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.664949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:37891 (local address 127.29.230.62:38923)
0810 15:16:45.664995 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.664997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.665206 (+ 209us) server_negotiation.cc:407] Connection header received
0810 15:16:45.665718 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.665719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.665730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.665781 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:45.666705 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.667243 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.668268 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.668468 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.668524 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.670580 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:45.670582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:45.670582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:45.670588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:45.671830 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.672215 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.672216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.672216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.672268 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.672456 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.672456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.672457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.672550 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:45.672596 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.672801 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.672862 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:16:45.673084 16760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.664917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38923 (local address 127.29.230.2:37891)
0810 15:16:45.665074 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.665113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.665626 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.665837 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.665840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.666418 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:45.666625 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.666630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.667363 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.667364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.668193 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.668195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.668543 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.668544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.668639 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.669156 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:45.669165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:45.670510 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:45.671911 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.671918 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.671920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.672145 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.672318 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.672318 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.672318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.672411 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.672592 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:45.672595 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:45.672706 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.672783 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.672901 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260810 15:16:45.673553 16756 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38923
I20260810 15:16:45.673657 16756 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:45.673833 16756 heartbeater.cc:511] Master 127.29.230.62:38923 requested a full tablet report, sending...
I20260810 15:16:45.674041 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 16539
I20260810 15:16:45.674145 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:16:45.674283 16218 ts_manager.cc:194] Registered new tserver with Master: 878cdc49d1f2445b8bb08fc9b01732e9 (127.29.230.2:46721)
I20260810 15:16:45.674347 30616 external_mini_cluster.cc:1428] Setting key d9f42463692413d81f153b6053e70bd7
I20260810 15:16:45.674717 16218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:37891
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:16:45.698201 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:16:45.778811 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:16:45.779031 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:16:45.779104 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:16:45.780408 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:16:45.780452 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:16:45.780485 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:16:45.781982 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:38923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:16:45.782220 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:16:45.782435 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:16:45.787204 16772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.787230 16775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:16:45.787245 16771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:16:45.787424 16765 server_base.cc:1034] running on GCE node
I20260810 15:16:45.787604 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:16:45.787813 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:16:45.788973 16765 hybrid_clock.cc:648] HybridClock initialized: now 1786375005788959 us; error 33 us; skew 500 ppm
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:16:45.791435 16765 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:16:45.791860 16765 webserver.cc:460] Webserver started at http://127.29.230.3:43795/ using document root <none> and password file <none>
I20260810 15:16:45.792063 16765 fs_manager.cc:359] Metadata directory not provided
I20260810 15:16:45.792107 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:16:45.792225 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:16:45.793287 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "d0b931a5bd9344abbf0f2a663fec60e5"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "beb06e93312ae388d1c8d627049adb0e"
server_key_iv: "3b90464636d908bd542aa9c37027bf27"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.793628 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "d0b931a5bd9344abbf0f2a663fec60e5"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "beb06e93312ae388d1c8d627049adb0e"
server_key_iv: "3b90464636d908bd542aa9c37027bf27"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.794965 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:16:45.795737 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.795930 16765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:16:45.796025 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "d0b931a5bd9344abbf0f2a663fec60e5"
format_stamp: "Formatted at 2026-08-10 15:16:45 on dist-test-slave-gjrz"
server_key: "beb06e93312ae388d1c8d627049adb0e"
server_key_iv: "3b90464636d908bd542aa9c37027bf27"
server_key_version: "encryptionkey@0"
I20260810 15:16:45.796082 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:16:45.813822 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:16:45.815946 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:16:45.816066 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:16:45.816263 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:16:45.816543 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:16:45.816574 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.816596 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:16:45.816610 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:16:45.822239 16765 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:39615
I20260810 15:16:45.822283 16985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:39615 every 8 connection(s)
I20260810 15:16:45.822605 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:16:45.823216 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 16765
I20260810 15:16:45.823319 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:16:45.823529 30616 external_mini_cluster.cc:1428] Setting key 949a44b91b00c9a2fbe2fc0d2eb0f124
Aug 10 15:16:45 dist-test-slave-gjrz krb5kdc[16143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375005, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:16:45.831459 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.823351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:37249 (local address 127.29.230.62:38923)
0810 15:16:45.823412 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.823414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.823678 (+ 264us) server_negotiation.cc:407] Connection header received
0810 15:16:45.824306 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.824307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.824316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.824351 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:45.825339 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.825847 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.826904 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.827107 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.827159 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.829243 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:45.829244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:45.829244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:45.829251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:45.830317 (+ 1066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.830664 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.830665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.830665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.830708 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.830880 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.830881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.830881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.830974 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:45.831048 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.831326 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.831394 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:16:45.831667 16989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.823331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:38923 (local address 127.29.230.3:37249)
0810 15:16:45.823518 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.823582 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.824216 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.824419 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.824423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.825052 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:16:45.825256 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.825261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.825939 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.825940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.826825 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.826828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.827167 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.827168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.827293 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.827763 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:16:45.827771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:16:45.829138 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:16:45.830383 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.830389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.830391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.830609 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.830751 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:16:45.830751 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:16:45.830752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:16:45.830841 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:16:45.831050 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:16:45.831056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:16:45.831182 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.831309 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.831451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260810 15:16:45.832154 16986 heartbeater.cc:347] Connected to a master server at 127.29.230.62:38923
I20260810 15:16:45.832255 16986 heartbeater.cc:464] Registering TS with master...
I20260810 15:16:45.832432 16986 heartbeater.cc:511] Master 127.29.230.62:38923 requested a full tablet report, sending...
I20260810 15:16:45.832783 16217 ts_manager.cc:194] Registered new tserver with Master: d0b931a5bd9344abbf0f2a663fec60e5 (127.29.230.3:39615)
I20260810 15:16:45.833137 16217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:37249
I20260810 15:16:45.835215 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:16:45.842197 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.836441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.29.230.62:38923)
0810 15:16:45.836499 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.836501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.836594 (+ 93us) server_negotiation.cc:407] Connection header received
0810 15:16:45.836652 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.836653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.836662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.836707 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:16:45.837203 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.837679 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.838354 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.838556 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.838608 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.839729 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:16:45.839730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:16:45.839730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:16:45.839737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:16:45.841010 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.841310 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.841311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.841311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.841369 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:16:45.841532 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:16:45.841533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:16:45.841533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:16:45.841721 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:16:45.841822 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.842057 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.842133 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:16:45.843665 16217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:16:45.845182 16217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:16:45.851547 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.29.230.2:46721)
0810 15:16:45.847738 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.847742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.847926 (+ 184us) server_negotiation.cc:407] Connection header received
0810 15:16:45.847973 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.847975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.847991 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.848062 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:45.848286 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849376 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.850631 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851132 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851197 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851259 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.851420 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.851462 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260810 15:16:45.851693 17006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44739 (local address 127.0.0.1:33338)
0810 15:16:45.848162 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.848171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.848221 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.848423 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.848424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.848426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:45.848462 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.848463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849631 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.849632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.850504 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.850507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851488 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.851552 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851556 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.851601 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.851635 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":378,"thread_start_us":70,"threads_started":1}
I20260810 15:16:45.851818 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.29.230.3:39615)
0810 15:16:45.847897 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.847900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.847910 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:45.847949 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.847951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.847968 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.848063 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:45.848249 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849083 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.850485 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851197 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851619 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851684 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.851709 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.851748 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":74,"threads_started":1}
I20260810 15:16:45.851580 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39615 (local address 127.0.0.1:58774)
0810 15:16:45.847569 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.847607 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.847677 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.848092 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.848095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.848097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:45.848166 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.848168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849170 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.849171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.850387 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.850390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851323 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.851387 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851392 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.851420 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.851479 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":85,"threads_started":1}
I20260810 15:16:45.851574 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:46721 (local address 127.0.0.1:53892)
0810 15:16:45.847860 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.847868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.847906 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.848146 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.848147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.848150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:45.848208 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.848210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849480 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.849481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.850561 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.850563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851209 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.851319 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851335 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.851398 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.851455 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260810 15:16:45.852021 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.847514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.29.230.1:44739)
0810 15:16:45.847702 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.847707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.848277 (+ 570us) server_negotiation.cc:407] Connection header received
0810 15:16:45.848322 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.848325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.848339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.848413 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:45.848532 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.849548 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.850595 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.851344 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.851399 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.851456 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.851895 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.851938 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260810 15:16:45.852591 16421 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 (DEFAULT_TABLE table=test-table [id=676a8e1e35134bbab004e03539a107df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:45.852586 16869 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 (DEFAULT_TABLE table=test-table [id=676a8e1e35134bbab004e03539a107df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:45.852663 16633 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 (DEFAULT_TABLE table=test-table [id=676a8e1e35134bbab004e03539a107df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:16:45.852960 16869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad73fb205754b5ebfb1d1c59a6eb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:45.852972 16421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad73fb205754b5ebfb1d1c59a6eb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:45.852977 16633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad73fb205754b5ebfb1d1c59a6eb240. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:16:45.855543 17016 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Bootstrap starting.
I20260810 15:16:45.856154 17017 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Bootstrap starting.
I20260810 15:16:45.856294 17016 tablet_bootstrap.cc:654] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:45.856828 17017 tablet_bootstrap.cc:654] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:45.856864 17016 log.cc:824] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:45.857378 17017 log.cc:824] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:45.857676 17016 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: No bootstrap required, opened a new log
I20260810 15:16:45.857731 17018 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Bootstrap starting.
I20260810 15:16:45.857750 17016 ts_tablet_manager.cc:1397] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:45.858074 17017 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: No bootstrap required, opened a new log
I20260810 15:16:45.858137 17017 ts_tablet_manager.cc:1397] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:45.858325 17018 tablet_bootstrap.cc:654] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Neither blocks nor log segments found. Creating new log.
I20260810 15:16:45.858839 17018 log.cc:824] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:16:45.859639 17018 tablet_bootstrap.cc:492] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: No bootstrap required, opened a new log
I20260810 15:16:45.859709 17018 ts_tablet_manager.cc:1397] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:16:45.859903 17017 raft_consensus.cc:348] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.859860 17016 raft_consensus.cc:348] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.860028 17016 raft_consensus.cc:374] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:45.860028 17017 raft_consensus.cc:374] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:45.860061 17016 raft_consensus.cc:729] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878cdc49d1f2445b8bb08fc9b01732e9, State: Initialized, Role: FOLLOWER
I20260810 15:16:45.860062 17017 raft_consensus.cc:729] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d41f047f76e4317b13ac0fc0b91eff6, State: Initialized, Role: FOLLOWER
I20260810 15:16:45.860200 17017 consensus_queue.cc:260] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.860208 17016 consensus_queue.cc:260] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.860466 17016 ts_tablet_manager.cc:1428] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:16:45.860491 17017 ts_tablet_manager.cc:1428] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:16:45.860590 16756 heartbeater.cc:503] Master 127.29.230.62:38923 was elected leader, sending a full tablet report...
I20260810 15:16:45.860601 16530 heartbeater.cc:503] Master 127.29.230.62:38923 was elected leader, sending a full tablet report...
I20260810 15:16:45.861510 17018 raft_consensus.cc:348] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.861656 17018 raft_consensus.cc:374] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:16:45.861687 17018 raft_consensus.cc:729] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b931a5bd9344abbf0f2a663fec60e5, State: Initialized, Role: FOLLOWER
I20260810 15:16:45.861789 17018 consensus_queue.cc:260] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.862047 17018 ts_tablet_manager.cc:1428] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:16:45.862074 16986 heartbeater.cc:503] Master 127.29.230.62:38923 was elected leader, sending a full tablet report...
W20260810 15:16:45.914836 16759 tablet.cc:2369] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:45.983000 17025 raft_consensus.cc:482] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:16:45.983137 17025 raft_consensus.cc:504] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.983542 17025 leader_election.cc:288] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 878cdc49d1f2445b8bb08fc9b01732e9 (127.29.230.2:46721), d0b931a5bd9344abbf0f2a663fec60e5 (127.29.230.3:39615)
I20260810 15:16:45.987412 17012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.983871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:42113 (local address 127.29.230.3:39615)
0810 15:16:45.983940 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.983942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.984043 (+ 101us) server_negotiation.cc:407] Connection header received
0810 15:16:45.984074 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.984075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.984085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.984150 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:45.984281 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.984999 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.986465 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.987016 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.987094 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.987105 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.987300 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.987342 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:16:45.987412 17007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.983816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:33073 (local address 127.29.230.2:46721)
0810 15:16:45.983868 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:45.983870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:45.983880 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:16:45.983978 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:45.983979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:45.983989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:45.984033 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:16:45.984226 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.984999 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.986458 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.987081 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.987155 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.987164 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:45.987298 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 15:16:45.987341 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:16:45.987425 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.983732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:46721 (local address 127.29.230.1:33073)
0810 15:16:45.983811 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.983824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.983893 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.984086 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.984087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.984089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:45.984161 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.984163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.985083 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.985084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.986348 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.986350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.987150 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.987150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.987220 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.987222 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.987291 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.987332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":86}
I20260810 15:16:45.987432 17027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:45.983732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:39615 (local address 127.29.230.1:42113)
0810 15:16:45.983935 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:45.983944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:16:45.983999 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:16:45.984158 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:16:45.984159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:16:45.984160 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:16:45.984215 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.984216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.985092 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.985093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.986348 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:16:45.986350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:16:45.987123 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:16:45.987124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:16:45.987215 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:45.987219 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:16:45.987291 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:16:45.987332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":73,"thread_start_us":76,"threads_started":1}
I20260810 15:16:45.987813 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad73fb205754b5ebfb1d1c59a6eb240" candidate_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" is_pre_election: true
I20260810 15:16:45.987988 16669 raft_consensus.cc:2454] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d41f047f76e4317b13ac0fc0b91eff6 in term 0.
I20260810 15:16:45.987988 16912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad73fb205754b5ebfb1d1c59a6eb240" candidate_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" is_pre_election: true
I20260810 15:16:45.988163 16912 raft_consensus.cc:2454] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d41f047f76e4317b13ac0fc0b91eff6 in term 0.
I20260810 15:16:45.988222 16341 leader_election.cc:302] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d41f047f76e4317b13ac0fc0b91eff6, 878cdc49d1f2445b8bb08fc9b01732e9; no voters:
I20260810 15:16:45.988353 17025 raft_consensus.cc:2781] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:16:45.988399 17025 raft_consensus.cc:482] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:16:45.988436 17025 raft_consensus.cc:3037] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:45.989153 17025 raft_consensus.cc:504] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.989277 17025 leader_election.cc:288] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [CANDIDATE]: Term 1 election: Requested vote from peers 878cdc49d1f2445b8bb08fc9b01732e9 (127.29.230.2:46721), d0b931a5bd9344abbf0f2a663fec60e5 (127.29.230.3:39615)
I20260810 15:16:45.989445 16912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad73fb205754b5ebfb1d1c59a6eb240" candidate_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b931a5bd9344abbf0f2a663fec60e5"
I20260810 15:16:45.989456 16669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad73fb205754b5ebfb1d1c59a6eb240" candidate_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878cdc49d1f2445b8bb08fc9b01732e9"
I20260810 15:16:45.989526 16669 raft_consensus.cc:3037] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:45.989527 16912 raft_consensus.cc:3037] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:16:45.990195 16669 raft_consensus.cc:2454] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d41f047f76e4317b13ac0fc0b91eff6 in term 1.
I20260810 15:16:45.990235 16912 raft_consensus.cc:2454] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d41f047f76e4317b13ac0fc0b91eff6 in term 1.
I20260810 15:16:45.990358 16341 leader_election.cc:302] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d41f047f76e4317b13ac0fc0b91eff6, 878cdc49d1f2445b8bb08fc9b01732e9; no voters:
I20260810 15:16:45.990468 17025 raft_consensus.cc:2781] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:16:45.990700 17025 raft_consensus.cc:686] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [term 1 LEADER]: Becoming Leader. State: Replica: 7d41f047f76e4317b13ac0fc0b91eff6, State: Running, Role: LEADER
I20260810 15:16:45.990849 17025 consensus_queue.cc:237] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } }
I20260810 15:16:45.991700 16218 catalog_manager.cc:5495] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d41f047f76e4317b13ac0fc0b91eff6 (127.29.230.1). New cstate: current_term: 1 leader_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d41f047f76e4317b13ac0fc0b91eff6" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44739 } health_report { overall_health: HEALTHY } } }
W20260810 15:16:46.010162 16531 tablet.cc:2369] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:16:46.031734 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:46.029213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.29.230.1:44739)
0810 15:16:46.029279 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:16:46.029281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:16:46.029290 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:16:46.029334 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:16:46.029334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:16:46.029360 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:16:46.029438 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:16:46.029585 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:46.030142 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:46.030941 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:16:46.031136 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:16:46.031192 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:16:46.031387 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:16:46.031504 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:16:46.031581 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:16:46.031604 (+ 23us) server_negotiation.cc:299] Negotiation successful
0810 15:16:46.031627 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:16:46.033653 16669 raft_consensus.cc:1261] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 1 FOLLOWER]: Refusing update from remote peer 7d41f047f76e4317b13ac0fc0b91eff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:46.033659 16912 raft_consensus.cc:1261] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 1 FOLLOWER]: Refusing update from remote peer 7d41f047f76e4317b13ac0fc0b91eff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:16:46.033993 17025 consensus_queue.cc:1035] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "878cdc49d1f2445b8bb08fc9b01732e9" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:46.034277 17028 consensus_queue.cc:1035] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 7d41f047f76e4317b13ac0fc0b91eff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b931a5bd9344abbf0f2a663fec60e5" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:16:46.036729 17033 mvcc.cc:205] Tried to move back new op lower bound from 7316992024711086080 to 7316992024539136000. Current Snapshot: MvccSnapshot[applied={T|T < 7316992024711086080}]
I20260810 15:16:46.036989 17034 mvcc.cc:205] Tried to move back new op lower bound from 7316992024711086080 to 7316992024539136000. Current Snapshot: MvccSnapshot[applied={T|T < 7316992024711086080}]
I20260810 15:16:46.037174 17035 mvcc.cc:205] Tried to move back new op lower bound from 7316992024711086080 to 7316992024539136000. Current Snapshot: MvccSnapshot[applied={T|T < 7316992024711086080}]
I20260810 15:16:46.039417 16217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:46.039506 16217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:16:46.040225 16217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75491e79b654a079f05260b07c783b2: Sending DeleteTablet for 3 replicas of tablet 2ad73fb205754b5ebfb1d1c59a6eb240
I20260810 15:16:46.040570 16869 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:46 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:58774
I20260810 15:16:46.040702 16421 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:46 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:33338
I20260810 15:16:46.040796 16633 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad73fb205754b5ebfb1d1c59a6eb240 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:16:46 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:53892
I20260810 15:16:46.040871 17053 tablet_replica.cc:330] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: stopping tablet replica
I20260810 15:16:46.041031 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 16310
I20260810 15:16:46.051151 17055 tablet_replica.cc:330] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: stopping tablet replica
I20260810 15:16:46.051312 17055 raft_consensus.cc:2229] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:46.051429 17055 raft_consensus.cc:2258] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:46.051870 17055 ts_tablet_manager.cc:1905] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:46.051918 17053 raft_consensus.cc:2229] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:16:46.052080 17053 raft_consensus.cc:2258] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:16:46.053861 17055 ts_tablet_manager.cc:1918] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:46.053923 17055 log.cc:1196] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/2ad73fb205754b5ebfb1d1c59a6eb240
I20260810 15:16:46.054076 17053 ts_tablet_manager.cc:1905] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:16:46.054124 17055 ts_tablet_manager.cc:1939] T 2ad73fb205754b5ebfb1d1c59a6eb240 P 878cdc49d1f2445b8bb08fc9b01732e9: Deleting consensus metadata
I20260810 15:16:46.054497 16193 catalog_manager.cc:4841] TS 878cdc49d1f2445b8bb08fc9b01732e9 (127.29.230.2:46721): tablet 2ad73fb205754b5ebfb1d1c59a6eb240 (table test-table [id=676a8e1e35134bbab004e03539a107df]) successfully deleted
I20260810 15:16:46.056063 17053 ts_tablet_manager.cc:1918] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:16:46.056144 17053 log.cc:1196] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/2ad73fb205754b5ebfb1d1c59a6eb240
I20260810 15:16:46.056388 17053 ts_tablet_manager.cc:1939] T 2ad73fb205754b5ebfb1d1c59a6eb240 P d0b931a5bd9344abbf0f2a663fec60e5: Deleting consensus metadata
I20260810 15:16:46.056850 16194 catalog_manager.cc:4841] TS d0b931a5bd9344abbf0f2a663fec60e5 (127.29.230.3:39615): tablet 2ad73fb205754b5ebfb1d1c59a6eb240 (table test-table [id=676a8e1e35134bbab004e03539a107df]) successfully deleted
W20260810 15:16:46.059978 16192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:44739 (error 108)
I20260810 15:16:46.060236 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 16539
I20260810 15:16:46.077314 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:16:46.077139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44739 (local address 127.0.0.1:33366)
0810 15:16:46.077219 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 15:16:46.077273 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 15:16:46.077502 16192 catalog_manager.cc:4584] TS 7d41f047f76e4317b13ac0fc0b91eff6 (127.29.230.1:44739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ad73fb205754b5ebfb1d1c59a6eb240: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44739: connect: Connection refused (error 111)
I20260810 15:16:46.080670 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 16765
I20260810 15:16:46.090718 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 16161
2026-08-10T15:16:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2242 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:16:46 dist-test-slave-gjrz krb5kdc[17059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:16:46 dist-test-slave-gjrz krb5kdc[17059](info): set up 2 sockets
Aug 10 15:16:46 dist-test-slave-gjrz krb5kdc[17059](info): commencing operation
krb5kdc: starting...
W20260810 15:16:47.280267 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:16:47 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:16:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangeradmin_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangeradmin_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangerlookup_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangerlookup_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
I20260810 15:16:47.439852 30616 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX3gT9X/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/postgres -l logfile start
2026-08-10 15:16:49.866 UTC [17103] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 15:16:49.866 UTC [17103] LOG: listening on IPv4 address "127.29.230.20", port 53539
2026-08-10 15:16:49.869 UTC [17103] LOG: listening on Unix socket "/tmp/.s.PGSQL.53539"
2026-08-10 15:16:49.890 UTC [17108] LOG: database system was shut down at 2026-08-10 15:16:48 UTC
2026-08-10 15:16:49.893 UTC [17103] LOG: database system is ready to accept connections
I20260810 15:16:50.981356 30616 mini_postgres.cc:96] Postgres bound to 53539
2026-08-10 15:16:50.989 UTC [17121] FATAL: database "slave" does not exist
127.29.230.20:53539 - accepting connections
I20260810 15:16:50.989822 30616 mini_ranger.cc:161] Starting Ranger...
I20260810 15:16:51.050736 30616 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 15:16:51.474139 30616 mini_ranger.cc:87] Created ranger Postgres database
I20260810 15:16:51.474210 30616 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin
2026-08-10 15:16:51,805 [I] DB FLAVOR :POSTGRES
2026-08-10 15:16:51,805 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:16:51,805 [I] Checking connection..
2026-08-10 15:16:51,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 15:16:52,070 [I] Checking connection passed.
2026-08-10 15:16:52,070 [I] --------- Verifying version history table ---------
2026-08-10 15:16:52,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:16:52,334 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:16:52,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:16:52,599 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:16:52,600 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 15:16:52,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 15:16:52.827 UTC [17226] WARNING: there is no transaction in progress
2026-08-10 15:16:52,836 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 15:16:52,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:16:53,094 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 15:16:53,094 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:16:53,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 15:16:53,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 15:16:53,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gjrz.c.gcp-upstream.internal','N') ;"
2026-08-10 15:16:53,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:16:54,074 [I] Table x_portal_user does not exist in database ranger
2026-08-10 15:16:54,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:16:54,321 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 15:16:54,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:16:54,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:16:54,844 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 15:16:54,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 15:16:55.793 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.808 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.823 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.838 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.853 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.871 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.922 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.929 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.942 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.952 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.962 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.975 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.985 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.991 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:55.998 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:56.005 UTC [17388] WARNING: there is no transaction in progress
2026-08-10 15:16:56,635 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 15:16:56,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 15:16:56,875 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 15:16:56,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:16:57,134 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 15:16:57,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:16:57,396 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 15:16:57,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:16:57,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:16:57,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gjrz.c.gcp-upstream.internal';"
2026-08-10 15:16:58,122 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 15:16:58,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 15:16:58,382 [I] DB_PATCHES have already been applied
I20260810 15:16:58.388432 30616 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX3gT9X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 15:16:58.388497 30616 mini_ranger.cc:193] Using host: 127.29.230.20
I20260810 15:16:58.392259 30616 mini_ranger.cc:227] Ranger admin URL: http://127.29.230.20:39509
Aug 10, 2026 3:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:16:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:16:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:16:59 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 15:16:59.634405 30616 mini_ranger.cc:160] Time spent starting Ranger: real 8.645s user 0.001s sys 0.010s
Aug 10, 2026 3:17:19 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:17:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 10, 2026 3:17:19 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:17:19 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:17:20 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 15:17:20.693684 30616 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangerkms_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/rangerkms_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.230.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.230.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 15:17:20 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375040, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 15:17:20.775745 30616 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 15:17:20.785400 30616 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 15:17:21.661530 30616 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 15:17:21.661612 30616 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms
I20260810 15:17:21.661628 30616 mini_ranger_kms.cc:227] Using postgres at 127.29.230.20:53539
2026-08-10 15:17:21,860 [I] DB FLAVOR :POSTGRES
2026-08-10 15:17:21,860 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:17:21,860 [I] Checking connection
2026-08-10 15:17:21,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 15:17:22,135 [I] connection success
2026-08-10 15:17:22,136 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 15:17:22,136 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 15:17:22,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 15:17:22,410 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 15:17:22,410 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:17:22,410 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 15:17:22,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:53539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 15:17:22,692 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 15:17:23.161559 30616 mini_ranger_kms.cc:310] Created kms service
I20260810 15:17:23.370697 30616 mini_ranger_kms.cc:326] Created kudu user
I20260810 15:17:23.413758 30616 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 15:17:23 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375040, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:23.895686 30616 mini_ranger_kms.cc:384] Added ranger policy
I20260810 15:17:23.895967 30616 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 15:17:23.895992 30616 mini_ranger_kms.cc:240] Using host: 127.29.230.20
I20260810 15:17:23.900254 30616 mini_ranger_kms.cc:276] Ranger KMS PID: 17707
I20260810 15:17:23.900398 30616 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.230.20:39325
Aug 10, 2026 3:17:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:17:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:17:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:17:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:17:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 15:17:26 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375046, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:26.462525 30616 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.687s user 0.004s sys 0.178s
I20260810 15:17:26.462649 30616 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 15:17:26.462738 30616 mini_ranger_kms.cc:401] 127.29.230.20:39325/kms/v1/keys
Aug 10 15:17:26 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375046, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
Aug 10 15:17:27 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375046, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
Aug 10 15:17:29 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:17:29.498322 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42049
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42049
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:39325/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:17:29.584681 17791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:17:29.584897 17791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:17:29.584918 17791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:17:29.586091 17791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:17:29.586140 17791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:17:29.586153 17791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:17:29.586165 17791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:17:29.586175 17791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:17:29.587918 17791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.230.20:39325/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42049
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:17:29.588158 17791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:17:29.588407 17791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:17:29.593948 17801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:29.594048 17804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:29.593977 17791 server_base.cc:1034] running on GCE node
W20260810 15:17:29.593950 17802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:29.594476 17791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:17:29.594848 17791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:17:29.596035 17791 hybrid_clock.cc:648] HybridClock initialized: now 1786375049596019 us; error 41 us; skew 500 ppm
Aug 10 15:17:29 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:29.599116 17791 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:17:29.599696 17791 webserver.cc:460] Webserver started at http://127.29.230.62:35559/ using document root <none> and password file <none>
I20260810 15:17:29.599967 17791 fs_manager.cc:359] Metadata directory not provided
I20260810 15:17:29.600023 17791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:17:29.600160 17791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:17:29 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:29.653959 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "ab98f330c5534534a236c93617c89d89"
format_stamp: "Formatted at 2026-08-10 15:17:29 on dist-test-slave-gjrz"
server_key: "2191473d894472380d679db2d75b6e99"
server_key_iv: "72b489f6b6f4246ac88c6ad08d226b5c"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:29.654431 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "ab98f330c5534534a236c93617c89d89"
format_stamp: "Formatted at 2026-08-10 15:17:29 on dist-test-slave-gjrz"
server_key: "2191473d894472380d679db2d75b6e99"
server_key_iv: "72b489f6b6f4246ac88c6ad08d226b5c"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:29.656297 17791 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 15:17:29.671180 17819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:29.671449 17791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:17:29.671558 17791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "ab98f330c5534534a236c93617c89d89"
format_stamp: "Formatted at 2026-08-10 15:17:29 on dist-test-slave-gjrz"
server_key: "2191473d894472380d679db2d75b6e99"
server_key_iv: "72b489f6b6f4246ac88c6ad08d226b5c"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:29.671674 17791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:17:29.685391 17791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:17:29.687655 17791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:17:29.687822 17791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:17:29.692998 17791 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:42049
I20260810 15:17:29.693022 17906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:42049 every 8 connection(s)
I20260810 15:17:29.693480 17791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:17:29.694249 17907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:17:29.695073 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 17791
I20260810 15:17:29.695161 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:17:29.695364 30616 external_mini_cluster.cc:1428] Setting key 0bbb6d17a36e5812274db798fd7144b3
I20260810 15:17:29.697507 17907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: Bootstrap starting.
I20260810 15:17:29.698244 17907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: Neither blocks nor log segments found. Creating new log.
I20260810 15:17:29.698549 17907 log.cc:824] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: Log is configured to *not* fsync() on all Append() calls
I20260810 15:17:29.699317 17907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: No bootstrap required, opened a new log
I20260810 15:17:29.701530 17907 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } }
I20260810 15:17:29.701670 17907 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:17:29.701692 17907 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab98f330c5534534a236c93617c89d89, State: Initialized, Role: FOLLOWER
I20260810 15:17:29.701800 17907 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } }
I20260810 15:17:29.701874 17907 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:17:29.701918 17907 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:17:29.701964 17907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 15:17:29 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:17:29.702504 17907 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } }
I20260810 15:17:29.702610 17907 leader_election.cc:302] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab98f330c5534534a236c93617c89d89; no voters:
I20260810 15:17:29.702766 17907 leader_election.cc:288] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:17:29.702811 17912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:17:29.702955 17912 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [term 1 LEADER]: Becoming Leader. State: Replica: ab98f330c5534534a236c93617c89d89, State: Running, Role: LEADER
I20260810 15:17:29.703063 17907 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:17:29.703091 17912 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } }
I20260810 15:17:29.703433 17912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab98f330c5534534a236c93617c89d89. Latest consensus state: current_term: 1 leader_uuid: "ab98f330c5534534a236c93617c89d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } } }
I20260810 15:17:29.703688 17912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [sys.catalog]: This master's current role is: LEADER
I20260810 15:17:29.703617 17919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab98f330c5534534a236c93617c89d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98f330c5534534a236c93617c89d89" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42049 } } }
I20260810 15:17:29.704099 17919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89 [sys.catalog]: This master's current role is: LEADER
I20260810 15:17:29.704156 17930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:17:29.704730 17907 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:17:29.704735 17930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:17:29.709563 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:29.696010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.29.230.62:42049)
0810 15:17:29.696215 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:29.696225 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:29.696258 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:17:29.696727 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:29.696737 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:29.696804 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:29.696956 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:17:29.697783 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:29.699046 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:29.699856 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:29.700489 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:29.700633 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:29.702965 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:17:29.702970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:17:29.702970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:17:29.702983 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:17:29.704895 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:29.705400 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:29.705403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:29.705403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:29.707239 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:29.707648 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:29.707650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:29.707650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:29.708458 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:17:29.708645 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:29.709201 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 15:17:29.709398 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260810 15:17:29.710538 17930 catalog_manager.cc:1347] Generated new cluster ID: 4304f182abae4c0fbe96f99d4f3e8cbb
I20260810 15:17:29.710592 17930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:17:29.717458 17930 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:17:29.718158 17930 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:17:29.724404 17930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: Generated new TSK 0
I20260810 15:17:29.724620 17930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:17:30.352239 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:39325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:17:30.445595 17980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:17:30.445788 17980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:17:30.445819 17980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:17:30.446985 17980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:17:30.447028 17980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:17:30.447062 17980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:17:30.448607 17980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.230.20:39325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:17:30.448841 17980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:17:30.449090 17980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:17:30.453866 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:30.454000 17992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:30.454181 17990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:30.454396 17980 server_base.cc:1034] running on GCE node
I20260810 15:17:30.454602 17980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:17:30.454839 17980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:17:30.456025 17980 hybrid_clock.cc:648] HybridClock initialized: now 1786375050456012 us; error 38 us; skew 500 ppm
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:30.458771 17980 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:17:30.459247 17980 webserver.cc:460] Webserver started at http://127.29.230.1:34987/ using document root <none> and password file <none>
I20260810 15:17:30.459510 17980 fs_manager.cc:359] Metadata directory not provided
I20260810 15:17:30.459559 17980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:17:30.459695 17980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:30.473374 17980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "1b18b24d733018a8f0d91800776f4de6"
server_key_iv: "4cd11a255ffdaa77013eedbf8e36d217"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:30.473770 17980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "1b18b24d733018a8f0d91800776f4de6"
server_key_iv: "4cd11a255ffdaa77013eedbf8e36d217"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:30.475958 17980 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260810 15:17:30.491740 18006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.491981 17980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:17:30.492092 17980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "1b18b24d733018a8f0d91800776f4de6"
server_key_iv: "4cd11a255ffdaa77013eedbf8e36d217"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:30.492204 17980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:17:30.513832 17980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:17:30.516350 17980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:17:30.516508 17980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:17:30.516808 17980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:17:30.517289 17980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:17:30.517438 17980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.517542 17980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:17:30.517632 17980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.527657 17980 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:33219
I20260810 15:17:30.528160 17980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:17:30.528501 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 17980
I20260810 15:17:30.528607 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:17:30.528780 30616 external_mini_cluster.cc:1428] Setting key 31329867591a3282daf3322a5d4567cc
I20260810 15:17:30.533409 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:33219 every 8 connection(s)
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:17:30.558552 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:30.535986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42049 (local address 127.29.230.1:48653)
0810 15:17:30.536213 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:30.536256 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:30.536897 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:30.547693 (+ 10796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:30.547698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:30.547901 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:17:30.548154 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:30.548160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.548928 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.548929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:30.549871 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:30.549874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.550251 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.550252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:30.550389 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:30.550956 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:17:30.550965 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:17:30.552557 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:17:30.556774 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:30.556782 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:30.556785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:30.557116 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:30.557365 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:30.557366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:30.557366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:30.557513 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:30.557824 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:17:30.557832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:17:30.558011 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 15:17:30.558120 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 15:17:30.558293 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
I20260810 15:17:30.559082 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:30.536190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:48653 (local address 127.29.230.62:42049)
0810 15:17:30.547398 (+ 11208us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:30.547400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:30.547413 (+ 13us) server_negotiation.cc:407] Connection header received
0810 15:17:30.547442 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:30.547443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:30.547455 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:30.547546 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:17:30.548265 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.548825 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.549960 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.550171 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.550229 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:30.552682 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:17:30.552684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:17:30.552684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:17:30.552693 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:17:30.556663 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:30.557215 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:30.557216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:30.557217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:30.557288 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:30.557592 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:30.557593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:30.557593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:30.557744 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:17:30.558842 (+ 1098us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:30.558889 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 15:17:30.558976 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11169,"thread_start_us":85,"threads_started":1}
I20260810 15:17:30.559497 18216 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42049
I20260810 15:17:30.559623 18216 heartbeater.cc:464] Registering TS with master...
I20260810 15:17:30.559850 18216 heartbeater.cc:511] Master 127.29.230.62:42049 requested a full tablet report, sending...
I20260810 15:17:30.560482 17847 ts_manager.cc:194] Registered new tserver with Master: 6aabfabc5ea54a2ab9b6701ba4a6b35d (127.29.230.1:33219)
I20260810 15:17:30.561232 17847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:48653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:17:30.568521 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:39325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:17:30.678560 18226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:17:30.678860 18226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:17:30.678903 18226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:17:30.680697 18226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:17:30.680760 18226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:17:30.680801 18226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:17:30.683192 18226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.230.20:39325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:17:30.683640 18226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:17:30.683964 18226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:17:30.690891 18245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:30.690914 18246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:30.691138 18226 server_base.cc:1034] running on GCE node
W20260810 15:17:30.691349 18248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:30.691649 18226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:17:30.691957 18226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:17:30.693127 18226 hybrid_clock.cc:648] HybridClock initialized: now 1786375050693099 us; error 40 us; skew 500 ppm
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:30.696496 18226 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:17:30.697044 18226 webserver.cc:460] Webserver started at http://127.29.230.2:41545/ using document root <none> and password file <none>
I20260810 15:17:30.697330 18226 fs_manager.cc:359] Metadata directory not provided
I20260810 15:17:30.697395 18226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:17:30.697538 18226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:30.719615 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "c318229e13fe4a9982d3171a2f0c3920"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "015960a953819bc9de043910a37b73b3"
server_key_iv: "166cb6e850fcb8a45e504578af4566fc"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:30.720167 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "c318229e13fe4a9982d3171a2f0c3920"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "015960a953819bc9de043910a37b73b3"
server_key_iv: "166cb6e850fcb8a45e504578af4566fc"
server_key_version: "kuduclusterkey@0"
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:30.727663 18226 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260810 15:17:30.770922 18261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.771154 18226 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.000s
I20260810 15:17:30.771256 18226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "c318229e13fe4a9982d3171a2f0c3920"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "015960a953819bc9de043910a37b73b3"
server_key_iv: "166cb6e850fcb8a45e504578af4566fc"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:30.771406 18226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:17:30.786825 18226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:17:30.788815 18226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:17:30.788955 18226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:17:30.789180 18226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:17:30.789551 18226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:17:30.789589 18226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.789619 18226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:17:30.789641 18226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:30.796629 18226 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:36667
I20260810 15:17:30.797052 18226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:17:30.797607 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:36667 every 8 connection(s)
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:17:30.805775 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 18226
I20260810 15:17:30.805884 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:17:30.805979 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:30.798168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:43245 (local address 127.29.230.62:42049)
0810 15:17:30.798212 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:30.798213 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:30.798224 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:17:30.798664 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:30.798666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:30.798683 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:30.798733 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:17:30.799326 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.799838 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.800823 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.801039 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.801094 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:30.803130 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:17:30.803132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:17:30.803133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:17:30.803141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:17:30.804550 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:30.804961 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:30.804963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:30.804963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:30.805027 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:30.805261 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:30.805262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:30.805262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:30.805382 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:17:30.805438 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:30.805782 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 15:17:30.805878 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:17:30.806078 30616 external_mini_cluster.cc:1428] Setting key 2b734a8379abb1e3f42e133a89515999
I20260810 15:17:30.806504 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:30.797826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42049 (local address 127.29.230.2:43245)
0810 15:17:30.797972 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:30.798015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:30.798560 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:30.798808 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:30.798813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:30.799011 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:17:30.799209 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:30.799214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.799926 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.799927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:30.800736 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:30.800739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:30.801112 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:30.801113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:30.801249 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:30.801713 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:17:30.801722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:17:30.803015 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:17:30.804643 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:30.804651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:30.804654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:30.804892 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:30.805085 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:30.805086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:30.805086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:30.805203 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:30.805439 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:17:30.805447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:17:30.805618 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 15:17:30.806101 (+ 483us) client_negotiation.cc:240] Negotiation successful
0810 15:17:30.806276 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260810 15:17:30.806973 18448 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42049
I20260810 15:17:30.807119 18448 heartbeater.cc:464] Registering TS with master...
I20260810 15:17:30.807395 18448 heartbeater.cc:511] Master 127.29.230.62:42049 requested a full tablet report, sending...
I20260810 15:17:30.807796 17847 ts_manager.cc:194] Registered new tserver with Master: c318229e13fe4a9982d3171a2f0c3920 (127.29.230.2:36667)
I20260810 15:17:30.808219 17847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:43245
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:17:30.835242 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:39325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:46401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:17:30.962044 18456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:17:30.962337 18456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:17:30.962373 18456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:17:30.964236 18456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:17:30.964303 18456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:17:30.964342 18456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:17:30.966521 18456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:46401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.230.20:39325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:42049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:17:30.966787 18456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:17:30.967056 18456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:17:30.973762 18464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:30.974046 18466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:17:30.974264 18462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:17:30.975332 18456 server_base.cc:1034] running on GCE node
I20260810 15:17:30.975602 18456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:17:30.975864 18456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:17:30.977027 18456 hybrid_clock.cc:648] HybridClock initialized: now 1786375050977003 us; error 36 us; skew 500 ppm
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:17:30.980367 18456 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:17:30.980882 18456 webserver.cc:460] Webserver started at http://127.29.230.3:44307/ using document root <none> and password file <none>
I20260810 15:17:30.981127 18456 fs_manager.cc:359] Metadata directory not provided
I20260810 15:17:30.981185 18456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:17:30.981314 18456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:17:30 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:30.999156 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "4d172a058ba64f4d9942d0ac19e43983"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "7106aeb4f469b97e688b50144c7312de"
server_key_iv: "7c3190319cf2943bed7fcb1dd2670ca0"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:31.000208 18456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "4d172a058ba64f4d9942d0ac19e43983"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "7106aeb4f469b97e688b50144c7312de"
server_key_iv: "7c3190319cf2943bed7fcb1dd2670ca0"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:31.002624 18456 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:17:31.020876 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:31.021075 18456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:17:31.021169 18456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "4d172a058ba64f4d9942d0ac19e43983"
format_stamp: "Formatted at 2026-08-10 15:17:30 on dist-test-slave-gjrz"
server_key: "7106aeb4f469b97e688b50144c7312de"
server_key_iv: "7c3190319cf2943bed7fcb1dd2670ca0"
server_key_version: "kuduclusterkey@0"
I20260810 15:17:31.021277 18456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:17:31.058008 18456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:17:31.060485 18456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:17:31.060629 18456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:17:31.060871 18456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:17:31.061237 18456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:17:31.061280 18456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:31.061308 18456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:17:31.061331 18456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:17:31.068645 18456 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36245
I20260810 15:17:31.069077 18456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:17:31.070231 18699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36245 every 8 connection(s)
I20260810 15:17:31.075524 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 18456
I20260810 15:17:31.075636 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:17:31.075830 30616 external_mini_cluster.cc:1428] Setting key 5b2c849ede43935442a17a3e665938f4
Aug 10 15:17:31 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:17:31.080469 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.069891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:38621 (local address 127.29.230.62:42049)
0810 15:17:31.069957 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.069960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.070204 (+ 244us) server_negotiation.cc:407] Connection header received
0810 15:17:31.070992 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.070994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.071006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.071068 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:17:31.071841 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.072632 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.074087 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.074403 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.074487 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.077249 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:17:31.077250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:17:31.077250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:17:31.077258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:17:31.078874 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:31.079405 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:31.079406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:31.079407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:31.079478 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:31.079736 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:31.079737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:31.079737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:31.079866 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:17:31.079912 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.080263 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.080361 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:17:31.081013 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.069884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42049 (local address 127.29.230.3:38621)
0810 15:17:31.070063 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.070114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.070861 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.071155 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.071160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.071444 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:17:31.071739 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.071746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.072733 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.072735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.073931 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.073934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.074502 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.074504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.074674 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.075314 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:17:31.075324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:17:31.077142 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:17:31.078977 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:31.078984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:31.078987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:31.079317 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:31.079542 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:17:31.079543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:17:31.079543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:17:31.079672 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:17:31.079942 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:17:31.079948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:17:31.080115 (+ 167us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.080607 (+ 492us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.080779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260810 15:17:31.081446 18701 heartbeater.cc:347] Connected to a master server at 127.29.230.62:42049
I20260810 15:17:31.081573 18701 heartbeater.cc:464] Registering TS with master...
I20260810 15:17:31.081782 18701 heartbeater.cc:511] Master 127.29.230.62:42049 requested a full tablet report, sending...
I20260810 15:17:31.082175 17847 ts_manager.cc:194] Registered new tserver with Master: 4d172a058ba64f4d9942d0ac19e43983 (127.29.230.3:36245)
I20260810 15:17:31.082671 17847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:38621
I20260810 15:17:31.095212 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:17:31.105196 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.096954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.29.230.62:42049)
0810 15:17:31.097003 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.097005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.097302 (+ 297us) server_negotiation.cc:407] Connection header received
0810 15:17:31.097401 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.097402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.097413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.097455 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:17:31.098307 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.099081 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.100196 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.100506 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.100550 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.102114 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:17:31.102116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:17:31.102116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:17:31.102123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:17:31.103879 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:31.104316 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:31.104318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:31.104318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:31.104377 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:17:31.104590 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:17:31.104591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:17:31.104591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:17:31.104723 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:17:31.104795 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.105014 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.105098 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:17:31.107414 17847 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:17:31.109748 17847 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:17:31.119982 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.113270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36667 (local address 127.0.0.1:35558)
0810 15:17:31.114802 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.114810 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.114849 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.115134 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.115138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.115139 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:17:31.115205 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.115206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.116872 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.116873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.118220 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.118223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.119671 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.119672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.119756 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.119784 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.119855 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.119908 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1519,"thread_start_us":71,"threads_started":1}
I20260810 15:17:31.120533 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.113523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.29.230.2:36667)
0810 15:17:31.114758 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.114763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.114894 (+ 131us) server_negotiation.cc:407] Connection header received
0810 15:17:31.114936 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.114939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.114954 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.115036 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:17:31.115308 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.116782 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.118625 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.119496 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.119554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.119638 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.120410 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.120451 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":84,"threads_started":1}
I20260810 15:17:31.121410 18340 tablet_service.cc:1467] Processing CreateTablet for tablet edaf302dd85b408ea6cf0c4152f1548f (DEFAULT_TABLE table=test-table [id=7664852fa86b4aefade4d9aeef0a1247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:17:31.121771 18340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf302dd85b408ea6cf0c4152f1548f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:17:31.126185 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.113791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36245 (local address 127.0.0.1:52358)
0810 15:17:31.119395 (+ 5604us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.119406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.119467 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.119759 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.119760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.119762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:17:31.119798 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.119799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.123407 (+ 3608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.123408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.124808 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.124811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.125932 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.125933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.125996 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.126001 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.126070 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.126111 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5583,"thread_start_us":66,"threads_started":1}
I20260810 15:17:31.126613 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.114051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.29.230.3:36245)
0810 15:17:31.114454 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.114459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.119541 (+ 5082us) server_negotiation.cc:407] Connection header received
0810 15:17:31.119588 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.119591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.119607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.119679 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:17:31.119878 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.120900 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.124914 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.125748 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.125817 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.125895 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.126484 (+ 589us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.126533 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":95,"threads_started":1}
I20260810 15:17:31.127575 18582 tablet_service.cc:1467] Processing CreateTablet for tablet edaf302dd85b408ea6cf0c4152f1548f (DEFAULT_TABLE table=test-table [id=7664852fa86b4aefade4d9aeef0a1247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:17:31.127978 18582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf302dd85b408ea6cf0c4152f1548f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:17:31.130952 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.112904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:33219 (local address 127.0.0.1:42954)
0810 15:17:31.114514 (+ 1610us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.114555 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.114617 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.119608 (+ 4991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.119610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.119613 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:17:31.119678 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.119680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.128601 (+ 8921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.128602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.129703 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.129707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.130716 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.130718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.130798 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.130803 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.130834 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.130875 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"thread_start_us":179,"threads_started":1}
I20260810 15:17:31.131393 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.113023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.29.230.1:33219)
0810 15:17:31.119362 (+ 6339us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.119367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.119379 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:17:31.119416 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.119418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.119435 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.119532 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:17:31.127334 (+ 7802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.128483 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.129822 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.130610 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.131148 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.131222 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.131250 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.131314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6300,"thread_start_us":74,"threads_started":1}
I20260810 15:17:31.132061 18095 tablet_service.cc:1467] Processing CreateTablet for tablet edaf302dd85b408ea6cf0c4152f1548f (DEFAULT_TABLE table=test-table [id=7664852fa86b4aefade4d9aeef0a1247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:17:31.132428 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edaf302dd85b408ea6cf0c4152f1548f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:17:31.136164 18737 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: Bootstrap starting.
I20260810 15:17:31.136907 18737 tablet_bootstrap.cc:654] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: Neither blocks nor log segments found. Creating new log.
I20260810 15:17:31.137238 18737 log.cc:824] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:17:31.138047 18737 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: No bootstrap required, opened a new log
I20260810 15:17:31.138128 18737 ts_tablet_manager.cc:1397] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:17:31.138317 18735 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Bootstrap starting.
I20260810 15:17:31.139463 18735 tablet_bootstrap.cc:654] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Neither blocks nor log segments found. Creating new log.
I20260810 15:17:31.139724 18734 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Bootstrap starting.
I20260810 15:17:31.140697 18737 raft_consensus.cc:348] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.140875 18737 raft_consensus.cc:374] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:17:31.140918 18737 raft_consensus.cc:729] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aabfabc5ea54a2ab9b6701ba4a6b35d, State: Initialized, Role: FOLLOWER
I20260810 15:17:31.141083 18737 consensus_queue.cc:260] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.141384 18737 ts_tablet_manager.cc:1428] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:17:31.141511 18216 heartbeater.cc:503] Master 127.29.230.62:42049 was elected leader, sending a full tablet report...
I20260810 15:17:31.140841 18734 tablet_bootstrap.cc:654] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Neither blocks nor log segments found. Creating new log.
I20260810 15:17:31.144321 18734 log.cc:824] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Log is configured to *not* fsync() on all Append() calls
I20260810 15:17:31.144934 18735 log.cc:824] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Log is configured to *not* fsync() on all Append() calls
I20260810 15:17:31.148553 18734 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: No bootstrap required, opened a new log
I20260810 15:17:31.148641 18734 ts_tablet_manager.cc:1397] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 15:17:31.150667 18734 raft_consensus.cc:348] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.150856 18734 raft_consensus.cc:374] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:17:31.150895 18734 raft_consensus.cc:729] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c318229e13fe4a9982d3171a2f0c3920, State: Initialized, Role: FOLLOWER
I20260810 15:17:31.151348 18734 consensus_queue.cc:260] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.151686 18734 ts_tablet_manager.cc:1428] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:17:31.151813 18448 heartbeater.cc:503] Master 127.29.230.62:42049 was elected leader, sending a full tablet report...
I20260810 15:17:31.153221 18735 tablet_bootstrap.cc:492] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: No bootstrap required, opened a new log
I20260810 15:17:31.153307 18735 ts_tablet_manager.cc:1397] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260810 15:17:31.155326 18735 raft_consensus.cc:348] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.155500 18735 raft_consensus.cc:374] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:17:31.155550 18735 raft_consensus.cc:729] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d172a058ba64f4d9942d0ac19e43983, State: Initialized, Role: FOLLOWER
I20260810 15:17:31.156668 18735 consensus_queue.cc:260] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.156985 18735 ts_tablet_manager.cc:1428] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 15:17:31.157105 18701 heartbeater.cc:503] Master 127.29.230.62:42049 was elected leader, sending a full tablet report...
I20260810 15:17:31.244575 18741 raft_consensus.cc:482] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:17:31.244751 18741 raft_consensus.cc:504] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.245211 18741 leader_election.cc:288] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c318229e13fe4a9982d3171a2f0c3920 (127.29.230.2:36667), 4d172a058ba64f4d9942d0ac19e43983 (127.29.230.3:36245)
I20260810 15:17:31.249469 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.245697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47013 (local address 127.29.230.3:36245)
0810 15:17:31.245923 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.245925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.245932 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:17:31.245956 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.245957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.245965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.246008 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:17:31.246253 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.247004 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.248502 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.249059 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.249111 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.249121 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.249348 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.249394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260810 15:17:31.249750 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.245412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36245 (local address 127.29.230.1:47013)
0810 15:17:31.245815 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.245822 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.245854 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.246063 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.246065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.246067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:17:31.246160 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.246162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.247133 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.247134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.248395 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.248398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.249184 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.249185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.249262 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.249266 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.249616 (+ 350us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.249662 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":110,"threads_started":1}
I20260810 15:17:31.250128 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf302dd85b408ea6cf0c4152f1548f" candidate_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d172a058ba64f4d9942d0ac19e43983" is_pre_election: true
I20260810 15:17:31.250360 18616 raft_consensus.cc:2454] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aabfabc5ea54a2ab9b6701ba4a6b35d in term 0.
Aug 10 15:17:31 dist-test-slave-gjrz krb5kdc[17059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375050, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:17:31.250785 18009 leader_election.cc:302] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d172a058ba64f4d9942d0ac19e43983, 6aabfabc5ea54a2ab9b6701ba4a6b35d; no voters:
I20260810 15:17:31.250946 18741 raft_consensus.cc:2781] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:17:31.251005 18741 raft_consensus.cc:482] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:17:31.251034 18741 raft_consensus.cc:3037] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:17:31.251870 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.245646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:36501 (local address 127.29.230.2:36667)
0810 15:17:31.245744 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.245746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.245781 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:17:31.245811 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.245812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.245821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.245864 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:17:31.246484 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.247240 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.250043 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.250561 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.250610 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.250620 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.251755 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.251804 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 15:17:31.251978 18741 raft_consensus.cc:504] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.252128 18741 leader_election.cc:288] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [CANDIDATE]: Term 1 election: Requested vote from peers c318229e13fe4a9982d3171a2f0c3920 (127.29.230.2:36667), 4d172a058ba64f4d9942d0ac19e43983 (127.29.230.3:36245)
I20260810 15:17:31.252342 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.245352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:36667 (local address 127.29.230.1:36501)
0810 15:17:31.245676 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.245687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:17:31.245742 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:17:31.246296 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:17:31.246298 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:17:31.246300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:17:31.246386 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.246388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.247375 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.247376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.248548 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:17:31.248551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.251588 (+ 3037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.251589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:17:31.251669 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.251673 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:17:31.252211 (+ 538us) client_negotiation.cc:240] Negotiation successful
0810 15:17:31.252257 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":111,"threads_started":1}
I20260810 15:17:31.252318 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf302dd85b408ea6cf0c4152f1548f" candidate_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d172a058ba64f4d9942d0ac19e43983"
I20260810 15:17:31.252393 18616 raft_consensus.cc:3037] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:17:31.253213 18369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf302dd85b408ea6cf0c4152f1548f" candidate_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c318229e13fe4a9982d3171a2f0c3920"
I20260810 15:17:31.253305 18616 raft_consensus.cc:2454] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aabfabc5ea54a2ab9b6701ba4a6b35d in term 1.
I20260810 15:17:31.253340 18369 raft_consensus.cc:3037] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:17:31.253481 18009 leader_election.cc:302] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d172a058ba64f4d9942d0ac19e43983, 6aabfabc5ea54a2ab9b6701ba4a6b35d; no voters:
I20260810 15:17:31.253574 18741 raft_consensus.cc:2781] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:17:31.253729 18741 raft_consensus.cc:686] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [term 1 LEADER]: Becoming Leader. State: Replica: 6aabfabc5ea54a2ab9b6701ba4a6b35d, State: Running, Role: LEADER
I20260810 15:17:31.253825 18741 consensus_queue.cc:237] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } }
I20260810 15:17:31.254172 18369 raft_consensus.cc:2454] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aabfabc5ea54a2ab9b6701ba4a6b35d in term 1.
I20260810 15:17:31.254323 18370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edaf302dd85b408ea6cf0c4152f1548f" candidate_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c318229e13fe4a9982d3171a2f0c3920" is_pre_election: true
I20260810 15:17:31.254382 18370 raft_consensus.cc:2362] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6aabfabc5ea54a2ab9b6701ba4a6b35d in term 1. Re-sending same reply.
I20260810 15:17:31.254705 17847 catalog_manager.cc:5495] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aabfabc5ea54a2ab9b6701ba4a6b35d (127.29.230.1). New cstate: current_term: 1 leader_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aabfabc5ea54a2ab9b6701ba4a6b35d" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 33219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 } health_report { overall_health: UNKNOWN } } }
I20260810 15:17:31.270272 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.266047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.29.230.1:33219)
0810 15:17:31.266089 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:17:31.266091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:17:31.266098 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:17:31.266136 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:17:31.266137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:17:31.266160 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:17:31.266222 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:17:31.266433 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.267095 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.269051 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:17:31.269326 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:17:31.269390 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:17:31.269640 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:17:31.269798 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:17:31.269883 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:17:31.270136 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 15:17:31.270164 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:17:31.276082 18616 raft_consensus.cc:1261] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 1 FOLLOWER]: Refusing update from remote peer 6aabfabc5ea54a2ab9b6701ba4a6b35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:17:31.276541 18752 consensus_queue.cc:1035] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d172a058ba64f4d9942d0ac19e43983" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:17:31.276815 18370 raft_consensus.cc:1261] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 1 FOLLOWER]: Refusing update from remote peer 6aabfabc5ea54a2ab9b6701ba4a6b35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:17:31.277143 18741 consensus_queue.cc:1035] T edaf302dd85b408ea6cf0c4152f1548f P 6aabfabc5ea54a2ab9b6701ba4a6b35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c318229e13fe4a9982d3171a2f0c3920" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:17:31.281817 18756 mvcc.cc:205] Tried to move back new op lower bound from 7316992210022735872 to 7316992209936306176. Current Snapshot: MvccSnapshot[applied={T|T < 7316992210022735872}]
I20260810 15:17:31.282104 18762 mvcc.cc:205] Tried to move back new op lower bound from 7316992210022735872 to 7316992209936306176. Current Snapshot: MvccSnapshot[applied={T|T < 7316992210022735872}]
I20260810 15:17:31.285120 17847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:17:31.285229 17847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:17:31.286191 17847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab98f330c5534534a236c93617c89d89: Sending DeleteTablet for 3 replicas of tablet edaf302dd85b408ea6cf0c4152f1548f
I20260810 15:17:31.286614 18095 tablet_service.cc:1514] Processing DeleteTablet for tablet edaf302dd85b408ea6cf0c4152f1548f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:17:31 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:42954
I20260810 15:17:31.286847 18340 tablet_service.cc:1514] Processing DeleteTablet for tablet edaf302dd85b408ea6cf0c4152f1548f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:17:31 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:35558
I20260810 15:17:31.287252 18582 tablet_service.cc:1514] Processing DeleteTablet for tablet edaf302dd85b408ea6cf0c4152f1548f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:17:31 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:52358
I20260810 15:17:31.287958 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 17980
I20260810 15:17:31.294070 18774 tablet_replica.cc:330] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: stopping tablet replica
I20260810 15:17:31.294209 18774 raft_consensus.cc:2229] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:17:31.294288 18774 pending_rounds.cc:62] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Trying to abort 2 pending ops.
I20260810 15:17:31.294313 18774 pending_rounds.cc:68] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7316992210022735872 op_type: WRITE_OP write_request { tablet_id: "edaf302dd85b408ea6cf0c4152f1548f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\267\331\347\323\006\"8\n\ntest-admin\022*\n 7664852fa86b4aefade4d9aeef0a1247\020\001\030\001 \001(\001" signature: "<redacted>""\316\342\010\\\370\364\243O\025\222Jd,0\271K\357~\035\036_\313\241EBT\371c\302q\372\373W\246\244\323\207\375&27\240\005\247\014\212\245G\307z\276\274\233u0\322\274\340\013\255^\322Q<" signing_key_seq_num: 0 } } request_id { client_id: "87a508e6f1644a7eb2d3be842493f90c" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260810 15:17:31.294463 18774 pending_rounds.cc:68] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7316992209936306176 op_type: NO_OP noop_request { }
I20260810 15:17:31.294502 18774 raft_consensus.cc:2866] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260810 15:17:31.294524 18774 raft_consensus.cc:2258] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:17:31.297974 18773 tablet_replica.cc:330] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: stopping tablet replica
I20260810 15:17:31.298228 18773 raft_consensus.cc:2229] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:17:31.298352 18773 raft_consensus.cc:2258] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:17:31.302181 18773 ts_tablet_manager.cc:1905] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:17:31.304353 18773 ts_tablet_manager.cc:1918] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:17:31.304486 18773 log.cc:1196] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/edaf302dd85b408ea6cf0c4152f1548f
I20260810 15:17:31.304876 18773 ts_tablet_manager.cc:1939] T edaf302dd85b408ea6cf0c4152f1548f P c318229e13fe4a9982d3171a2f0c3920: Deleting consensus metadata
I20260810 15:17:31.307613 17823 catalog_manager.cc:4841] TS c318229e13fe4a9982d3171a2f0c3920 (127.29.230.2:36667): tablet edaf302dd85b408ea6cf0c4152f1548f (table test-table [id=7664852fa86b4aefade4d9aeef0a1247]) successfully deleted
W20260810 15:17:31.310926 18758 consensus_queue.cc:1159] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d172a058ba64f4d9942d0ac19e43983" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
W20260810 15:17:31.311241 18487 connection.cc:441] server connection from 127.29.230.1:47013 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.29.230.1:47013 (request call id 3) could send its response
I20260810 15:17:31.311848 18774 ts_tablet_manager.cc:1905] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 15:17:31.312183 17823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:33219 (error 108)
I20260810 15:17:31.312889 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 18226
I20260810 15:17:31.315740 18774 ts_tablet_manager.cc:1918] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:17:31.315821 18774 log.cc:1196] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/edaf302dd85b408ea6cf0c4152f1548f
I20260810 15:17:31.316448 18774 ts_tablet_manager.cc:1939] T edaf302dd85b408ea6cf0c4152f1548f P 4d172a058ba64f4d9942d0ac19e43983: Deleting consensus metadata
I20260810 15:17:31.316972 17820 catalog_manager.cc:4841] TS 4d172a058ba64f4d9942d0ac19e43983 (127.29.230.3:36245): tablet edaf302dd85b408ea6cf0c4152f1548f (table test-table [id=7664852fa86b4aefade4d9aeef0a1247]) successfully deleted
I20260810 15:17:31.320827 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:17:31.320676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:33219 (local address 127.0.0.1:42976)
0810 15:17:31.320739 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:17:31.320786 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 15:17:31.321252 17823 catalog_manager.cc:4584] TS 6aabfabc5ea54a2ab9b6701ba4a6b35d (127.29.230.1:33219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edaf302dd85b408ea6cf0c4152f1548f: Network error: Client connection negotiation failed: client connection to 127.29.230.1:33219: connect: Connection refused (error 111)
I20260810 15:17:31.331090 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 18456
I20260810 15:17:31.370471 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 17791
I20260810 15:17:31.385582 30616 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 15:17:31.731899 30616 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 15:17:31.731999 30616 mini_ranger.cc:66] Stopping Ranger...
I20260810 15:17:32.076710 30616 mini_ranger.cc:68] Stopped Ranger
2026-08-10 15:17:32.076 UTC [17103] LOG: received smart shutdown request
2026-08-10 15:17:32.080 UTC [17103] LOG: background worker "logical replication launcher" (PID 17119) exited with exit code 1
2026-08-10 15:17:32.080 UTC [17109] LOG: shutting down
2026-08-10 15:17:32.250 UTC [17103] LOG: database system is shut down
2026-08-10T15:17:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46244 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:17:32 dist-test-slave-gjrz krb5kdc[18790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:17:32 dist-test-slave-gjrz krb5kdc[18790](info): set up 2 sockets
Aug 10 15:17:32 dist-test-slave-gjrz krb5kdc[18790](info): commencing operation
krb5kdc: starting...
W20260810 15:17:33.527331 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:17:33 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:17:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangeradmin_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangeradmin_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangerlookup_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangerlookup_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
I20260810 15:17:33.697690 30616 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX3gT9X/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/postgres -l logfile start
2026-08-10 15:17:36.075 UTC [18837] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 15:17:36.075 UTC [18837] LOG: listening on IPv4 address "127.29.230.20", port 57613
2026-08-10 15:17:36.079 UTC [18837] LOG: listening on Unix socket "/tmp/.s.PGSQL.57613"
2026-08-10 15:17:36.100 UTC [18841] LOG: database system was shut down at 2026-08-10 15:17:34 UTC
2026-08-10 15:17:36.104 UTC [18837] LOG: database system is ready to accept connections
I20260810 15:17:37.194059 30616 mini_postgres.cc:96] Postgres bound to 57613
2026-08-10 15:17:37.200 UTC [18852] FATAL: database "slave" does not exist
127.29.230.20:57613 - accepting connections
I20260810 15:17:37.201323 30616 mini_ranger.cc:161] Starting Ranger...
I20260810 15:17:37.216004 30616 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 15:17:37.672403 30616 mini_ranger.cc:87] Created ranger Postgres database
I20260810 15:17:37.672487 30616 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin
2026-08-10 15:17:37,946 [I] DB FLAVOR :POSTGRES
2026-08-10 15:17:37,947 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:17:37,947 [I] Checking connection..
2026-08-10 15:17:37,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 15:17:38,186 [I] Checking connection passed.
2026-08-10 15:17:38,186 [I] --------- Verifying version history table ---------
2026-08-10 15:17:38,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:17:38,451 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:17:38,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:17:38,709 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 15:17:38,709 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 15:17:38,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 15:17:38.967 UTC [18971] WARNING: there is no transaction in progress
2026-08-10 15:17:38,975 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 15:17:38,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 15:17:39,218 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 15:17:39,218 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:17:39,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 15:17:39,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 15:17:39,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gjrz.c.gcp-upstream.internal','N') ;"
2026-08-10 15:17:39,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:17:40,242 [I] Table x_portal_user does not exist in database ranger
2026-08-10 15:17:40,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:17:40,493 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 15:17:40,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:17:40,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:17:40,989 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 15:17:40,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 15:17:41.955 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:41.970 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:41.985 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:41.999 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.015 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.034 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.097 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.103 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.116 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.125 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.135 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.149 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.158 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.164 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.170 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42.177 UTC [19141] WARNING: there is no transaction in progress
2026-08-10 15:17:42,773 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 15:17:42,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 15:17:43,002 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 15:17:43,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 15:17:43,246 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 15:17:43,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 15:17:43,503 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 15:17:43,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 15:17:43,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 15:17:43,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gjrz.c.gcp-upstream.internal';"
2026-08-10 15:17:44,247 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 15:17:44,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 15:17:44,506 [I] DB_PATCHES have already been applied
I20260810 15:17:44.511965 30616 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX3gT9X/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 15:17:44.512029 30616 mini_ranger.cc:193] Using host: 127.29.230.20
I20260810 15:17:44.515857 30616 mini_ranger.cc:227] Ranger admin URL: http://127.29.230.20:42103
Aug 10, 2026 3:17:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:17:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:17:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:17:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:17:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:17:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:17:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 15:17:45.826084 30616 mini_ranger.cc:160] Time spent starting Ranger: real 8.625s user 0.001s sys 0.008s
Aug 10, 2026 3:18:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:18:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:18:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:18:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:18:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 15:18:07.348336 30616 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.230.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.230.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangerkms_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.230.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/rangerkms_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.230.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.230.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.230.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/HTTP_127.29.230.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 15:18:07 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375087, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 15:18:07.445722 30616 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 15:18:07.455840 30616 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 15:18:08.259778 30616 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 15:18:08.259845 30616 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms
I20260810 15:18:08.259861 30616 mini_ranger_kms.cc:227] Using postgres at 127.29.230.20:57613
2026-08-10 15:18:08,416 [I] DB FLAVOR :POSTGRES
2026-08-10 15:18:08,416 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 15:18:08,416 [I] Checking connection
2026-08-10 15:18:08,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 15:18:08,680 [I] connection success
2026-08-10 15:18:08,681 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 15:18:08,681 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 15:18:08,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 15:18:08,915 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 15:18:08,915 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 15:18:08,916 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 15:18:08,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.230.20:57613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 15:18:09,187 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 15:18:09.651639 30616 mini_ranger_kms.cc:310] Created kms service
I20260810 15:18:09.869367 30616 mini_ranger_kms.cc:326] Created kudu user
I20260810 15:18:09.911603 30616 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 15:18:09 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375087, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:10.296226 30616 mini_ranger_kms.cc:384] Added ranger policy
I20260810 15:18:10.296342 30616 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX3gT9X/build/release/bin/postgresql.jar:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX3gT9X/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 15:18:10.296360 30616 mini_ranger_kms.cc:240] Using host: 127.29.230.20
I20260810 15:18:10.302721 30616 mini_ranger_kms.cc:276] Ranger KMS PID: 19456
I20260810 15:18:10.302793 30616 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.230.20:58375
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:18:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 15:18:12 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375092, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 15:18:12 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375092, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:12.909425 30616 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.464s user 0.007s sys 0.132s
I20260810 15:18:12.909519 30616 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 15:18:12.909566 30616 mini_ranger_kms.cc:401] 127.29.230.20:58375/kms/v1/keys
Aug 10 15:18:13 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375092, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.230.20@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
Aug 10 15:18:15 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:18:15.566805 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36837
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36837
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:15.649731 19554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:15.649931 19554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:15.649952 19554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:15.651099 19554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:15.651141 19554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:15.651154 19554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:15.651165 19554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:15.651176 19554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:15.652807 19554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.230.20:58375/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36837
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:15.653028 19554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:15.653246 19554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:15.658193 19561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:15.658320 19554 server_base.cc:1034] running on GCE node
W20260810 15:18:15.658205 19565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:15.658191 19560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:15.658779 19554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:15.659107 19554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:15.660260 19554 hybrid_clock.cc:648] HybridClock initialized: now 1786375095660217 us; error 53 us; skew 500 ppm
Aug 10 15:18:15 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375095, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:15.665736 19554 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:15.666244 19554 webserver.cc:460] Webserver started at http://127.29.230.62:33709/ using document root <none> and password file <none>
I20260810 15:18:15.666514 19554 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:15.666566 19554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:15.666693 19554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:18:15 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375095, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:15.715530 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "2aede64d27a441db9f2702044f670428"
format_stamp: "Formatted at 2026-08-10 15:18:15 on dist-test-slave-gjrz"
server_key: "1e61657bc92682cd46eeb88eb30e4e22"
server_key_iv: "7d42374810127a1be85ce076c175c62a"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:15.715937 19554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "2aede64d27a441db9f2702044f670428"
format_stamp: "Formatted at 2026-08-10 15:18:15 on dist-test-slave-gjrz"
server_key: "1e61657bc92682cd46eeb88eb30e4e22"
server_key_iv: "7d42374810127a1be85ce076c175c62a"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:15.717453 19554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:18:15.732831 19575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:15.733116 19554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:15.733242 19554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "2aede64d27a441db9f2702044f670428"
format_stamp: "Formatted at 2026-08-10 15:18:15 on dist-test-slave-gjrz"
server_key: "1e61657bc92682cd46eeb88eb30e4e22"
server_key_iv: "7d42374810127a1be85ce076c175c62a"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:15.733355 19554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:15.740239 19554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:15.742187 19554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:15.742341 19554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:15.747066 19554 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36837
I20260810 15:18:15.747131 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36837 every 8 connection(s)
I20260810 15:18:15.747502 19554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:15.748229 19669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:15.750571 19669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Bootstrap starting.
I20260810 15:18:15.751155 19669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:15.751452 19669 log.cc:824] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:15.752105 19669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: No bootstrap required, opened a new log
I20260810 15:18:15.753242 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 19554
I20260810 15:18:15.753345 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:18:15.753547 30616 external_mini_cluster.cc:1428] Setting key 344b4f51e30ca8e76cc492a499246408
I20260810 15:18:15.754341 19669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:15.754510 19669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:15.754545 19669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aede64d27a441db9f2702044f670428, State: Initialized, Role: FOLLOWER
I20260810 15:18:15.754704 19669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:15.754868 19669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:15.754922 19669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:15.754966 19669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:15.755730 19669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:15.755837 19669 leader_election.cc:302] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aede64d27a441db9f2702044f670428; no voters:
I20260810 15:18:15.755998 19669 leader_election.cc:288] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:18:15.756044 19679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:15.756234 19679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 LEADER]: Becoming Leader. State: Replica: 2aede64d27a441db9f2702044f670428, State: Running, Role: LEADER
I20260810 15:18:15.756345 19669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:15.756361 19679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:15.756748 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aede64d27a441db9f2702044f670428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } } }
I20260810 15:18:15.756891 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:15.757043 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aede64d27a441db9f2702044f670428. Latest consensus state: current_term: 1 leader_uuid: "2aede64d27a441db9f2702044f670428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } } }
I20260810 15:18:15.757160 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:15.758044 19669 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:18:15.758460 19692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:15.758957 19692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 10 15:18:15 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:15.764355 19719 mvcc.cc:205] Tried to move back new op lower bound from 7316992392244486144 to 7316992392218611712. Current Snapshot: MvccSnapshot[applied={T|T < 7316992392244486144}]
I20260810 15:18:15.765363 19692 catalog_manager.cc:1347] Generated new cluster ID: 2dd705ab6623404284c8718c17686ce0
I20260810 15:18:15.765442 19692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:18:15.767828 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:15.754210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.29.230.62:36837)
0810 15:18:15.754399 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:15.754409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:15.754442 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:18:15.754568 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:15.754575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:15.754634 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:15.754786 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:15.755604 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:15.757158 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:15.758268 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:15.758902 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:15.759025 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:15.762114 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:15.762170 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:15.762170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:15.762271 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:15.765268 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:15.765676 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:15.765680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:15.765680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:15.765816 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:15.766016 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:15.766017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:15.766017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:15.766397 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:15.766528 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:15.767085 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 15:18:15.767498 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260810 15:18:15.801570 19692 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:18:15.802547 19692 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:15.807482 19692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Generated new TSK 0
I20260810 15:18:15.807796 19692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:18:16.501955 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:16.588937 19761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:16.589143 19761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:16.589174 19761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:16.590297 19761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:16.590341 19761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:16.590375 19761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:16.591856 19761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:16.592093 19761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:16.592324 19761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:16.597149 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:16.597163 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:16.597262 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:16.597692 19761 server_base.cc:1034] running on GCE node
I20260810 15:18:16.597880 19761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:16.598114 19761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:16.599255 19761 hybrid_clock.cc:648] HybridClock initialized: now 1786375096599241 us; error 31 us; skew 500 ppm
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:16.601742 19761 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:16.602164 19761 webserver.cc:460] Webserver started at http://127.29.230.1:35019/ using document root <none> and password file <none>
I20260810 15:18:16.602365 19761 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:16.602413 19761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:16.602527 19761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:16.616015 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "d0ff4be8ac6f4f03be38bcce996b15df"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "271a74731f1486a893a2e8bc27994e5a"
server_key_iv: "1dbf862585bd7d89b6e62cb857c987e3"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.616606 19761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "d0ff4be8ac6f4f03be38bcce996b15df"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "271a74731f1486a893a2e8bc27994e5a"
server_key_iv: "1dbf862585bd7d89b6e62cb857c987e3"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.618494 19761 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:16.630276 19785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.630470 19761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:16.630573 19761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "d0ff4be8ac6f4f03be38bcce996b15df"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "271a74731f1486a893a2e8bc27994e5a"
server_key_iv: "1dbf862585bd7d89b6e62cb857c987e3"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.630666 19761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:16.645572 19761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:16.647446 19761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:16.647589 19761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:16.647840 19761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:16.648161 19761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:16.648196 19761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.648224 19761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:16.648249 19761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.655032 19761 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:45051
I20260810 15:18:16.655149 19976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:45051 every 8 connection(s)
I20260810 15:18:16.655531 19761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:18:16.658066 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 19761
I20260810 15:18:16.658164 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:18:16.658352 30616 external_mini_cluster.cc:1428] Setting key 0d305e59353eac82b988c2960db36470
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:16.668006 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:16.656689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:60311 (local address 127.29.230.62:36837)
0810 15:18:16.657386 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:16.657388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:16.657397 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:16.657425 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:16.657426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:16.657439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:16.657515 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:16.658610 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.661008 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.662458 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.662658 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.662723 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:16.665139 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:16.665140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:16.665141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:16.665147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:16.666498 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:16.666943 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:16.666943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:16.666944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:16.666993 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:16.667256 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:16.667257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:16.667257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:16.667410 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:16.667475 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:16.667853 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 15:18:16.667933 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:18:16.668520 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:16.656320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.1:60311)
0810 15:18:16.656507 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:16.656555 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:16.657166 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:16.657636 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:16.657642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:16.657913 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:16.658214 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:16.658221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.661109 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.661110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:16.662352 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:16.662356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.662760 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.662761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:16.662900 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:16.663495 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:16.663506 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:16.665036 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:16.666579 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:16.666587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:16.666589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:16.666871 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:16.667047 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:16.667048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:16.667049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:16.667195 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:16.667506 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:16.667513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:16.667683 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 15:18:16.668162 (+ 479us) client_negotiation.cc:240] Negotiation successful
0810 15:18:16.668322 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260810 15:18:16.669011 19978 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:16.669112 19978 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:16.669296 19978 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:16.669991 19604 ts_manager.cc:194] Registered new tserver with Master: d0ff4be8ac6f4f03be38bcce996b15df (127.29.230.1:45051)
I20260810 15:18:16.670939 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:60311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:18:16.690871 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:16.793974 19991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:16.794168 19991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:16.794198 19991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:16.795367 19991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:16.795408 19991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:16.795442 19991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:16.796967 19991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:16.797212 19991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:16.797430 19991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:16.802484 20003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:16.802449 20006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:16.802851 20004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:16.803241 19991 server_base.cc:1034] running on GCE node
I20260810 15:18:16.803476 19991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:16.803705 19991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:16.804843 19991 hybrid_clock.cc:648] HybridClock initialized: now 1786375096804832 us; error 30 us; skew 500 ppm
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:16.807317 19991 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:16.807794 19991 webserver.cc:460] Webserver started at http://127.29.230.2:37451/ using document root <none> and password file <none>
I20260810 15:18:16.808035 19991 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:16.808084 19991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:16.808240 19991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:16.831079 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "27818374448a4e968eb48de5548cf6db"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "e47406bb180d170d08d8f73aba6b0c5b"
server_key_iv: "471a001f20706322a94ddb6ad865e6d0"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.831561 19991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "27818374448a4e968eb48de5548cf6db"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "e47406bb180d170d08d8f73aba6b0c5b"
server_key_iv: "471a001f20706322a94ddb6ad865e6d0"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.833415 19991 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:16.849717 20019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.849905 19991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:16.850001 19991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "27818374448a4e968eb48de5548cf6db"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "e47406bb180d170d08d8f73aba6b0c5b"
server_key_iv: "471a001f20706322a94ddb6ad865e6d0"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:16.850103 19991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:16.879710 19991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:16.882391 19991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:16.882649 19991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:16.882979 19991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:16.883466 19991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:16.883574 19991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.883664 19991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:16.883725 19991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:16.891201 19991 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:44825
I20260810 15:18:16.891743 19991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:18:16.897836 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 19991
I20260810 15:18:16.897949 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:18:16.898145 30616 external_mini_cluster.cc:1428] Setting key ce5e2c9132273d2722f2dd1090412671
I20260810 15:18:16.900264 20210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:44825 every 8 connection(s)
Aug 10 15:18:16 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
I20260810 15:18:16.912259 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:16.901342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:46401 (local address 127.29.230.62:36837)
0810 15:18:16.901397 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:16.901399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:16.901543 (+ 144us) server_negotiation.cc:407] Connection header received
0810 15:18:16.902356 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:16.902357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:16.902373 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:16.902432 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:16.903154 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.903976 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.905503 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.905814 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.905862 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:16.908755 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:16.908756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:16.908757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:16.908765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:16.910398 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:16.910886 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:16.910888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:16.910888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:16.911150 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:16.911492 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:16.911493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:16.911493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:16.911631 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:16.911697 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:16.912054 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 15:18:16.912154 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:18:16.912447 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:16.901043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.2:46401)
0810 15:18:16.901424 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:16.901470 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:16.902220 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:16.902515 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:16.902520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:16.902772 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:16.903044 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:16.903049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.904091 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.904093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:16.905381 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:16.905385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:16.905979 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:16.905981 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:16.906161 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:16.906898 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:16.906909 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:16.908639 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:16.910508 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:16.910515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:16.910518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:16.910800 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:16.911238 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:16.911239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:16.911239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:16.911421 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:16.911699 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:16.911705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:16.911880 (+ 175us) client_negotiation.cc:769] Sending connection context
0810 15:18:16.912034 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 15:18:16.912208 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
I20260810 15:18:16.913096 20211 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:16.913200 20211 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:16.913393 20211 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:16.913849 19604 ts_manager.cc:194] Registered new tserver with Master: 27818374448a4e968eb48de5548cf6db (127.29.230.2:44825)
I20260810 15:18:16.914311 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:46401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:18:16.931644 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:17.066468 20218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:17.066828 20218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:17.066929 20218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:17.068753 20218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:17.068869 20218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:17.068974 20218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:17.071194 20218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:17.071686 20218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:17.072062 20218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:17.080046 20227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.080416 20228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.084682 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:17.085196 20218 server_base.cc:1034] running on GCE node
I20260810 15:18:17.085553 20218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:17.085841 20218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:17.087014 20218 hybrid_clock.cc:648] HybridClock initialized: now 1786375097086991 us; error 35 us; skew 500 ppm
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:17.090583 20218 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:17.091140 20218 webserver.cc:460] Webserver started at http://127.29.230.3:46235/ using document root <none> and password file <none>
I20260810 15:18:17.091434 20218 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:17.091491 20218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:17.091626 20218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:17.117327 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "e862e38deac4462daf60e1e8d07bc1cc"
format_stamp: "Formatted at 2026-08-10 15:18:17 on dist-test-slave-gjrz"
server_key: "fb6561db9de76b4ae6897381a1802dd3"
server_key_iv: "525ab3d638b3551cd8314776fa586be9"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.117812 20218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "e862e38deac4462daf60e1e8d07bc1cc"
format_stamp: "Formatted at 2026-08-10 15:18:17 on dist-test-slave-gjrz"
server_key: "fb6561db9de76b4ae6897381a1802dd3"
server_key_iv: "525ab3d638b3551cd8314776fa586be9"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.125443 20218 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260810 15:18:17.144852 20245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.145048 20218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 15:18:17.145156 20218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "e862e38deac4462daf60e1e8d07bc1cc"
format_stamp: "Formatted at 2026-08-10 15:18:17 on dist-test-slave-gjrz"
server_key: "fb6561db9de76b4ae6897381a1802dd3"
server_key_iv: "525ab3d638b3551cd8314776fa586be9"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.145256 20218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:17.213088 20218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:17.215586 20218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:17.215804 20218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:17.216125 20218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:17.216590 20218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:17.216686 20218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.216764 20218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:17.216825 20218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.225708 20218 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:41191
I20260810 15:18:17.226202 20218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:17.227665 20445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:41191 every 8 connection(s)
I20260810 15:18:17.234449 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 20218
I20260810 15:18:17.234553 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:17.234756 30616 external_mini_cluster.cc:1428] Setting key d14f4bf1b7cd4160cca359ab8baa07f9
I20260810 15:18:17.238492 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.227534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:54955 (local address 127.29.230.62:36837)
0810 15:18:17.227579 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:17.227581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:17.228021 (+ 440us) server_negotiation.cc:407] Connection header received
0810 15:18:17.228862 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:17.228864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:17.228877 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:17.228909 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:17.229785 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.230582 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.232135 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.232453 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.232496 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.235078 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:17.235079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:17.235080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:17.235088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:17.236821 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.237334 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.237335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.237336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.237399 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.237709 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.237710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.237711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.237853 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:17.237899 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:17.238291 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 15:18:17.238383 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:18:17.239032 20451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.227212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.3:54955)
0810 15:18:17.227888 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:17.227931 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:17.228728 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:17.229109 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:17.229114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:17.229372 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:17.229667 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.229674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.230699 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.230700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.232030 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.232033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.232601 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.232602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.232742 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.233323 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:17.233335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:17.234967 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:17.236936 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.236943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.236945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.237243 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.237488 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.237488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.237489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.237628 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.237950 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:17.237957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:17.238128 (+ 171us) client_negotiation.cc:769] Sending connection context
0810 15:18:17.238628 (+ 500us) client_negotiation.cc:240] Negotiation successful
0810 15:18:17.238797 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":102,"threads_started":1}
I20260810 15:18:17.239625 20449 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:17.239724 20449 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.239920 20449 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:17.240353 19604 ts_manager.cc:194] Registered new tserver with Master: e862e38deac4462daf60e1e8d07bc1cc (127.29.230.3:41191)
I20260810 15:18:17.240844 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:54955
I20260810 15:18:17.249617 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:18:17.250298 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 19761
I20260810 15:18:17.320827 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 19991
I20260810 15:18:17.341032 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 20218
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:17.358155 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 19554
I20260810 15:18:17.372768 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36837
--webserver_interface=127.29.230.62
--webserver_port=33709
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36837
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:17.454125 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:17.454334 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:17.454365 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:17.455528 20455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:17.455567 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:17.455585 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:17.455601 20455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:17.455615 20455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:17.457167 20455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36837
--ranger_config_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=33709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:17.457401 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:17.457600 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:17.462258 20462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.462270 20461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.462278 20466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:17.462343 20455 server_base.cc:1034] running on GCE node
I20260810 15:18:17.462672 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:17.462937 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:17.464066 20455 hybrid_clock.cc:648] HybridClock initialized: now 1786375097464061 us; error 36 us; skew 500 ppm
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:17.466482 20455 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:17.466899 20455 webserver.cc:460] Webserver started at http://127.29.230.62:33709/ using document root <none> and password file <none>
I20260810 15:18:17.467069 20455 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:17.467123 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:17.537783 20455 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:17.538609 20479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.538786 20455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:17.538903 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "2aede64d27a441db9f2702044f670428"
format_stamp: "Formatted at 2026-08-10 15:18:15 on dist-test-slave-gjrz"
server_key: "1e61657bc92682cd46eeb88eb30e4e22"
server_key_iv: "7d42374810127a1be85ce076c175c62a"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.539429 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:17.552466 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:17.553982 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:17.554098 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:17.558056 20455 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36837
I20260810 15:18:17.558094 20567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36837 every 8 connection(s)
I20260810 15:18:17.558449 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:17.558983 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 20455
I20260810 15:18:17.559231 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:45051
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:36837
--webserver_port=35019
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:17.559854 20568 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:18:17.560489 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Bootstrap starting.
I20260810 15:18:17.568221 20568 log.cc:824] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:17.569883 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:18:17.570142 20568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Bootstrap complete.
I20260810 15:18:17.572381 20568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:17.572613 20568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aede64d27a441db9f2702044f670428, State: Initialized, Role: FOLLOWER
I20260810 15:18:17.572779 20568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:17.572837 20568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:17.572882 20568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:17.572928 20568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:18:17.573897 20568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:17.573994 20568 leader_election.cc:302] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aede64d27a441db9f2702044f670428; no voters:
I20260810 15:18:17.574162 20568 leader_election.cc:288] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:18:17.574224 20575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:18:17.574410 20568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:17.574441 20575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [term 2 LEADER]: Becoming Leader. State: Replica: 2aede64d27a441db9f2702044f670428, State: Running, Role: LEADER
I20260810 15:18:17.574558 20575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } }
I20260810 15:18:17.574856 20575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aede64d27a441db9f2702044f670428. Latest consensus state: current_term: 2 leader_uuid: "2aede64d27a441db9f2702044f670428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } } }
I20260810 15:18:17.574915 20575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:17.574911 20578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2aede64d27a441db9f2702044f670428" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aede64d27a441db9f2702044f670428" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36837 } } }
I20260810 15:18:17.575204 20578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:17.575861 20568 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 15:18:17.576248 20592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:17.579053 20592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:17.579420 20592 catalog_manager.cc:1259] Loaded cluster ID: 2dd705ab6623404284c8718c17686ce0
I20260810 15:18:17.579635 20592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:18:17.581089 20592 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:17.581317 20592 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Loaded TSK: 0
I20260810 15:18:17.581629 20592 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:18:17.670039 20573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:17.670387 20573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:17.670467 20573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:17.672169 20573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:17.672231 20573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:17.672268 20573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:17.674388 20573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:45051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=35019
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:17.674752 20573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:17.675093 20573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:17.681762 20637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.682091 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.682305 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:17.682477 20573 server_base.cc:1034] running on GCE node
I20260810 15:18:17.682698 20573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:17.682921 20573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:17.684101 20573 hybrid_clock.cc:648] HybridClock initialized: now 1786375097684073 us; error 38 us; skew 500 ppm
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:17.687606 20573 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:17.688127 20573 webserver.cc:460] Webserver started at http://127.29.230.1:35019/ using document root <none> and password file <none>
I20260810 15:18:17.688385 20573 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:17.688443 20573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:17.719694 20573 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 15:18:17.721482 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.721706 20573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260810 15:18:17.721805 20573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "d0ff4be8ac6f4f03be38bcce996b15df"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "271a74731f1486a893a2e8bc27994e5a"
server_key_iv: "1dbf862585bd7d89b6e62cb857c987e3"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.722304 20573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:17.756182 20573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:17.758518 20573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:17.758726 20573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:17.759035 20573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:17.759446 20573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:17.759528 20573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.759621 20573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:17.759688 20573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.766417 20573 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:45051
I20260810 15:18:17.766598 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:45051 every 8 connection(s)
I20260810 15:18:17.767076 20573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:17.777017 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 20573
I20260810 15:18:17.777339 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:44825
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:36837
--webserver_port=37451
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:17.782657 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.768299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:47401 (local address 127.29.230.62:36837)
0810 15:18:17.769022 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:17.769031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:17.769069 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:18:17.769232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:17.769240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:17.769297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:17.769446 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:17.770278 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.771710 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.773265 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.773897 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.774493 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.776618 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:17.776623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:17.776623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:17.776635 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:17.780662 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.781212 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.781216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.781217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.781342 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.781651 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.781652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.781652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.781918 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:17.781997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:17.782307 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 15:18:17.782429 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":74,"threads_started":1}
I20260810 15:18:17.782673 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.767915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.1:47401)
0810 15:18:17.768123 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:17.768174 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:17.768802 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:17.769533 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:17.769538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:17.769766 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:17.770131 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.770138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.771832 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.771833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.773156 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.773159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.774044 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.774045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.774195 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.774804 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:17.774814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:17.776494 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:17.780752 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.780763 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.780766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.781104 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.781439 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.781440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.781440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.781585 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.781999 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:17.782006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:17.782145 (+ 139us) client_negotiation.cc:769] Sending connection context
0810 15:18:17.782241 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 15:18:17.782441 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260810 15:18:17.783953 20858 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:17.784060 20858 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.784238 20858 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:17.785056 20858 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:36837 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:18:17.785127 20858 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.785205 20858 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:17.785516 20858 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.785595 20858 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:17.785871 20858 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:18:17.885679 20863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:17.885876 20863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:17.885907 20863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:17.887164 20863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:17.887207 20863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:17.887240 20863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:17.888810 20863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:44825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=37451
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:17.889051 20863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:17.889271 20863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:17.894146 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:17.894289 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:17.894609 20863 server_base.cc:1034] running on GCE node
W20260810 15:18:17.895417 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:17.895743 20863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:17.896029 20863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:17.897199 20863 hybrid_clock.cc:648] HybridClock initialized: now 1786375097897179 us; error 35 us; skew 500 ppm
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:17.899718 20863 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:17.900166 20863 webserver.cc:460] Webserver started at http://127.29.230.2:37451/ using document root <none> and password file <none>
I20260810 15:18:17.900383 20863 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:17.900449 20863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:17.917863 20863 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:17.920365 20894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.920652 20863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:17.920763 20863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "27818374448a4e968eb48de5548cf6db"
format_stamp: "Formatted at 2026-08-10 15:18:16 on dist-test-slave-gjrz"
server_key: "e47406bb180d170d08d8f73aba6b0c5b"
server_key_iv: "471a001f20706322a94ddb6ad865e6d0"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:17.921226 20863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:17.931793 20863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:17.934178 20863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:17.934353 20863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:17.934654 20863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:17.935016 20863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:17.935065 20863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.935094 20863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:17.935133 20863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:17.942337 20863 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:44825
I20260810 15:18:17.942400 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:44825 every 8 connection(s)
I20260810 15:18:17.942833 20863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:18:17.945592 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 20863
I20260810 15:18:17.945842 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.230.20:58375/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:41191
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:36837
--webserver_port=46235
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 15:18:17 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375097, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:17.954026 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.943703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.2:49919)
0810 15:18:17.943893 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:17.943947 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:17.944607 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:17.944892 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:17.944896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:17.945131 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:17.945446 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.945454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.946484 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.946485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.947810 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:17.947814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.948227 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.948228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:17.948374 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.948960 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:17.948971 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:17.950602 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:17.952458 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.952465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.952468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.952761 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.952975 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:17.952976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:17.952976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:17.953101 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:17.953381 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:17.953386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:17.953551 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 15:18:17.953654 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 15:18:17.953801 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260810 15:18:17.954504 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:17.943695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:49919 (local address 127.29.230.62:36837)
0810 15:18:17.943742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:17.943744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:17.944048 (+ 304us) server_negotiation.cc:407] Connection header received
0810 15:18:17.944724 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:17.944725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:17.944739 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:17.944799 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:17.945590 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.946355 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.947918 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:17.948123 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:17.948175 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:17.950708 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:17.950709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:17.950710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:17.950718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:17.952366 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.952845 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.952847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.952847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.952909 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:17.953168 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:17.953169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:17.953169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:17.953302 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:17.953357 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:17.954333 (+ 976us) server_negotiation.cc:299] Negotiation successful
0810 15:18:17.954420 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:18:17.957254 21093 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:17.957397 21093 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.957638 21093 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:17.958278 21093 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:36837 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:18:17.958393 21093 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.958489 21093 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:17.958815 21093 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:17.958926 21093 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:17.959227 21093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:18:18.066486 21100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:18.066687 21100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:18.066718 21100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:18.067963 21100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:18.068014 21100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:18.068051 21100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:18.069633 21100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.230.20:58375/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:41191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=46235
--tserver_master_addrs=127.29.230.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:18.069859 21100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:18.070068 21100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:18.074949 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:18.074926 21115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:18.075215 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:18.075191 21100 server_base.cc:1034] running on GCE node
I20260810 15:18:18.075670 21100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:18.075915 21100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:18.077095 21100 hybrid_clock.cc:648] HybridClock initialized: now 1786375098077074 us; error 32 us; skew 500 ppm
Aug 10 15:18:18 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:18.079923 21100 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:18.080453 21100 webserver.cc:460] Webserver started at http://127.29.230.3:46235/ using document root <none> and password file <none>
I20260810 15:18:18.080737 21100 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:18.080811 21100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 15:18:18 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:18.097306 21100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:18.098533 21131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:18.098699 21100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:18:18.098783 21100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "e862e38deac4462daf60e1e8d07bc1cc"
format_stamp: "Formatted at 2026-08-10 15:18:17 on dist-test-slave-gjrz"
server_key: "fb6561db9de76b4ae6897381a1802dd3"
server_key_iv: "525ab3d638b3551cd8314776fa586be9"
server_key_version: "kuduclusterkey@0"
I20260810 15:18:18.099179 21100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:18.108752 21100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:18.110857 21100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:18.111006 21100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:18.111258 21100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:18.111644 21100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:18.111697 21100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:18.111738 21100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:18.111764 21100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:18.118108 21100 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:41191
I20260810 15:18:18.118203 21333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:41191 every 8 connection(s)
I20260810 15:18:18.118494 21100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:18.119390 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 21100
Aug 10 15:18:18 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:18.128324 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:18.119362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:53705 (local address 127.29.230.62:36837)
0810 15:18:18.119419 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:18.119422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:18.119590 (+ 168us) server_negotiation.cc:407] Connection header received
0810 15:18:18.120338 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:18.120339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:18.120349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:18.120400 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:18.120913 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.121408 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.122374 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.122641 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.122707 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:18.124783 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:18.124784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:18.124784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:18.124790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:18.126699 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:18.127185 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:18.127186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:18.127186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:18.127246 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:18.127574 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:18.127576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:18.127576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:18.127705 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:18.127747 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:18.128136 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 15:18:18.128217 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:18:18.128643 21342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:18.119224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:36837 (local address 127.29.230.3:53705)
0810 15:18:18.119475 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:18.119504 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:18.120245 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:18.120468 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:18.120472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:18.120645 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:18.120831 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:18.120835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.121486 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.121487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:18.122296 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:18.122298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.122793 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.122794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:18.122900 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:18.123361 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:18.123370 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:18.124696 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:18.126786 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:18.126793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:18.126795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:18.127097 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:18.127360 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:18.127360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:18.127361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:18.127492 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:18.127795 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:18.127801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:18.127961 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 15:18:18.128260 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 15:18:18.128417 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":98,"threads_started":1}
I20260810 15:18:18.128233 21345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:18.119742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.29.230.62:36837)
0810 15:18:18.120051 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:18.120053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:18.120060 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:18:18.120086 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:18.120087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:18.120098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:18.120151 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:18.120933 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.121695 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.122798 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:18.123117 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:18.123179 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:18.124683 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:18.124684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:18.124685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:18.124692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:18.126613 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:18.126999 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:18.127000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:18.127000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:18.127072 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:18.127284 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:18.127284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:18.127285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:18.127414 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:18.127559 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:18.128038 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 15:18:18.128124 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":68,"threads_started":1}
I20260810 15:18:18.129070 21339 heartbeater.cc:347] Connected to a master server at 127.29.230.62:36837
I20260810 15:18:18.129173 21339 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:18.129369 21339 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:18.130008 21339 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:36837 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 15:18:18.130085 21339 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:18.130155 21339 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:18.130474 21339 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:18.130556 21339 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
W20260810 15:18:18.130870 21339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 15:18:18 dist-test-slave-gjrz krb5kdc[18790](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:18.786056 20858 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:18.786234 20858 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:18.786676 20506 ts_manager.cc:194] Registered new tserver with Master: d0ff4be8ac6f4f03be38bcce996b15df (127.29.230.1:45051)
I20260810 15:18:18.787638 20506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:47401
Aug 10 15:18:18 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
I20260810 15:18:18.959436 21093 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:18.959578 21093 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:18.959875 20506 ts_manager.cc:194] Registered new tserver with Master: 27818374448a4e968eb48de5548cf6db (127.29.230.2:44825)
I20260810 15:18:18.960412 20506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:49919
I20260810 15:18:19.131050 21339 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:19.131217 21339 heartbeater.cc:511] Master 127.29.230.62:36837 requested a full tablet report, sending...
I20260810 15:18:19.131667 20506 ts_manager.cc:194] Registered new tserver with Master: e862e38deac4462daf60e1e8d07bc1cc (127.29.230.3:41191)
I20260810 15:18:19.132187 20506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:53705
I20260810 15:18:19.141341 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:18:19.150380 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.142564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.29.230.62:36837)
0810 15:18:19.142683 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.142685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.142827 (+ 142us) server_negotiation.cc:407] Connection header received
0810 15:18:19.142879 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.142880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.142894 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.142965 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:19.143766 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.144543 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.145593 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.145898 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.145962 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.147398 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:19.147399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:19.147400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:19.147407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:19.149148 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:19.149552 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:19.149553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:19.149553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:19.149622 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:19.149828 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:19.149829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:19.149829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:19.149964 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:19.150059 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.150192 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.150280 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
I20260810 15:18:19.152737 20506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:18:19.155143 20506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:18:19.166805 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.161832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.29.230.3:41191)
0810 15:18:19.162462 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.162466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.162633 (+ 167us) server_negotiation.cc:407] Connection header received
0810 15:18:19.162672 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.162675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.162690 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.162759 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:19.163027 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.164091 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.165559 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.166313 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.166381 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.166441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.166696 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.166734 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":95,"threads_started":1}
I20260810 15:18:19.166800 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.161252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:41191 (local address 127.0.0.1:56704)
0810 15:18:19.162525 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.162534 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:19.162582 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:19.162869 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:19.162872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:19.162875 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:19.162943 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.162945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.164191 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.164192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.165448 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.165452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.166493 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.166494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.166576 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.166591 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:18:19.166656 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:18:19.166714 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"thread_start_us":97,"threads_started":1}
I20260810 15:18:19.167865 21227 tablet_service.cc:1467] Processing CreateTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 (DEFAULT_TABLE table=test-table [id=7573ae0ac8e34927a9e8a3e16e86c213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:19.168277 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477ab2343c745b2aa0b25910cfbc146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:19.172291 21399 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Bootstrap starting.
I20260810 15:18:19.175550 21399 tablet_bootstrap.cc:654] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:19.175989 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.161513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45051 (local address 127.0.0.1:57168)
0810 15:18:19.166975 (+ 5462us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.166984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:19.167017 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:19.167294 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:19.167295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:19.167296 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:19.167327 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.167328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.172498 (+ 5170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.172499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.173553 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.173556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.175745 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.175746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.175809 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.175814 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:18:19.175859 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:18:19.175923 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5451,"thread_start_us":67,"threads_started":1}
I20260810 15:18:19.176126 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.161244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:44825 (local address 127.0.0.1:52954)
0810 15:18:19.162228 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.162270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:19.162337 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:19.171222 (+ 8885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:19.171223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:19.171226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:19.171324 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.171325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.173606 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.173607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.174871 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.174874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.175905 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.175906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.175972 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.175976 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:18:19.176004 (+ 28us) client_negotiation.cc:240] Negotiation successful
0810 15:18:19.176044 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":107,"threads_started":1}
I20260810 15:18:19.176498 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.161545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.29.230.2:44825)
0810 15:18:19.170796 (+ 9251us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.170800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.170812 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:18:19.170845 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.170847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.170861 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.170939 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:19.172490 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.173487 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.174965 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.175774 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.176289 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.176362 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.176387 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.176426 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9211,"thread_start_us":58,"threads_started":1}
I20260810 15:18:19.177004 21399 log.cc:824] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:19.177140 20952 tablet_service.cc:1467] Processing CreateTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 (DEFAULT_TABLE table=test-table [id=7573ae0ac8e34927a9e8a3e16e86c213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:19.177393 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.162029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.29.230.1:45051)
0810 15:18:19.162171 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.162176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.167071 (+ 4895us) server_negotiation.cc:407] Connection header received
0810 15:18:19.167110 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.167112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.167127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.167187 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:19.171387 (+ 4200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.172386 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.173654 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.174721 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.176518 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.177170 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.177201 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.177243 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260810 15:18:19.177484 20952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477ab2343c745b2aa0b25910cfbc146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:19.181424 20738 tablet_service.cc:1467] Processing CreateTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 (DEFAULT_TABLE table=test-table [id=7573ae0ac8e34927a9e8a3e16e86c213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:19.181793 20738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7477ab2343c745b2aa0b25910cfbc146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:19.185083 21406 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Bootstrap starting.
I20260810 15:18:19.185513 21405 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Bootstrap starting.
I20260810 15:18:19.185860 21406 tablet_bootstrap.cc:654] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:19.186133 21405 tablet_bootstrap.cc:654] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:19.186295 21406 log.cc:824] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:19.186426 21405 log.cc:824] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:19.187146 21406 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: No bootstrap required, opened a new log
I20260810 15:18:19.187218 21406 ts_tablet_manager.cc:1397] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:19.187245 21399 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: No bootstrap required, opened a new log
I20260810 15:18:19.187357 21399 ts_tablet_manager.cc:1397] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260810 15:18:19.189683 21406 raft_consensus.cc:348] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.189803 21406 raft_consensus.cc:374] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:19.189822 21406 raft_consensus.cc:729] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ff4be8ac6f4f03be38bcce996b15df, State: Initialized, Role: FOLLOWER
I20260810 15:18:19.189935 21399 raft_consensus.cc:348] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.190097 21399 raft_consensus.cc:374] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:19.190131 21399 raft_consensus.cc:729] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e862e38deac4462daf60e1e8d07bc1cc, State: Initialized, Role: FOLLOWER
I20260810 15:18:19.190243 21399 consensus_queue.cc:260] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.190438 21406 consensus_queue.cc:260] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.190546 21339 heartbeater.cc:503] Master 127.29.230.62:36837 was elected leader, sending a full tablet report...
I20260810 15:18:19.190742 21406 ts_tablet_manager.cc:1428] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 15:18:19.190840 20858 heartbeater.cc:503] Master 127.29.230.62:36837 was elected leader, sending a full tablet report...
I20260810 15:18:19.191038 21399 ts_tablet_manager.cc:1428] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260810 15:18:19.192448 21405 tablet_bootstrap.cc:492] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: No bootstrap required, opened a new log
I20260810 15:18:19.192530 21405 ts_tablet_manager.cc:1397] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 15:18:19.194924 21405 raft_consensus.cc:348] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.195082 21405 raft_consensus.cc:374] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:19.195114 21405 raft_consensus.cc:729] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27818374448a4e968eb48de5548cf6db, State: Initialized, Role: FOLLOWER
I20260810 15:18:19.195238 21405 consensus_queue.cc:260] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.195495 21405 ts_tablet_manager.cc:1428] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:18:19.195606 21093 heartbeater.cc:503] Master 127.29.230.62:36837 was elected leader, sending a full tablet report...
Aug 10 15:18:19 dist-test-slave-gjrz krb5kdc[18790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for HTTP/127.29.230.20@KRBTEST.COM
W20260810 15:18:19.268643 20859 tablet.cc:2369] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:18:19.369864 21341 tablet.cc:2369] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:18:19.444216 21095 tablet.cc:2369] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:18:19.496914 21413 raft_consensus.cc:482] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:19.497087 21413 raft_consensus.cc:504] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.497535 21413 leader_election.cc:288] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27818374448a4e968eb48de5548cf6db (127.29.230.2:44825), e862e38deac4462daf60e1e8d07bc1cc (127.29.230.3:41191)
I20260810 15:18:19.501935 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.497766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:41191 (local address 127.29.230.1:35631)
0810 15:18:19.498722 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.498733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:19.498822 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:19.499006 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:19.499006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:19.499008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:19.499090 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.499092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.500088 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.500089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.500927 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.500929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.501641 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.501716 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.501721 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:18:19.501780 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:18:19.501820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":98,"threads_started":1}
I20260810 15:18:19.501926 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.498040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:35631 (local address 127.29.230.3:41191)
0810 15:18:19.498303 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.498305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.498792 (+ 487us) server_negotiation.cc:407] Connection header received
0810 15:18:19.498897 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.498898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.498907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.498934 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:19.499173 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.499901 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501017 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.501534 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501609 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.501618 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.501794 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.501841 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260810 15:18:19.502136 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.498080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:42793 (local address 127.29.230.2:44825)
0810 15:18:19.498127 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.498151 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.498162 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:18:19.498193 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.498194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.498204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.498340 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:19.498687 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.499601 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501211 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.501741 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501801 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.501808 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.502032 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.502080 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:18:19.502290 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.497715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:44825 (local address 127.29.230.1:42793)
0810 15:18:19.498031 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.498044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:19.498107 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:19.498484 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:19.498486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:19.498488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:19.498587 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.498589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.499803 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.499804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.501095 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:19.501099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.501842 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.501843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:19.501929 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.501934 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:18:19.502199 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 15:18:19.502234 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":32,"thread_start_us":124,"threads_started":1}
I20260810 15:18:19.502731 21261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477ab2343c745b2aa0b25910cfbc146" candidate_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e862e38deac4462daf60e1e8d07bc1cc" is_pre_election: true
I20260810 15:18:19.502808 21020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477ab2343c745b2aa0b25910cfbc146" candidate_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27818374448a4e968eb48de5548cf6db" is_pre_election: true
I20260810 15:18:19.502887 21261 raft_consensus.cc:2454] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0ff4be8ac6f4f03be38bcce996b15df in term 0.
I20260810 15:18:19.502969 21020 raft_consensus.cc:2454] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0ff4be8ac6f4f03be38bcce996b15df in term 0.
I20260810 15:18:19.503132 20661 leader_election.cc:302] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0ff4be8ac6f4f03be38bcce996b15df, e862e38deac4462daf60e1e8d07bc1cc; no voters:
I20260810 15:18:19.503331 21413 raft_consensus.cc:2781] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:18:19.503376 21413 raft_consensus.cc:482] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:18:19.503398 21413 raft_consensus.cc:3037] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:19.504124 21413 raft_consensus.cc:504] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.504251 21413 leader_election.cc:288] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [CANDIDATE]: Term 1 election: Requested vote from peers 27818374448a4e968eb48de5548cf6db (127.29.230.2:44825), e862e38deac4462daf60e1e8d07bc1cc (127.29.230.3:41191)
I20260810 15:18:19.504405 21020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477ab2343c745b2aa0b25910cfbc146" candidate_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27818374448a4e968eb48de5548cf6db"
I20260810 15:18:19.504416 21261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7477ab2343c745b2aa0b25910cfbc146" candidate_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e862e38deac4462daf60e1e8d07bc1cc"
I20260810 15:18:19.504496 21261 raft_consensus.cc:3037] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:19.504495 21020 raft_consensus.cc:3037] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:19.505151 21261 raft_consensus.cc:2454] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0ff4be8ac6f4f03be38bcce996b15df in term 1.
I20260810 15:18:19.505336 20661 leader_election.cc:302] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0ff4be8ac6f4f03be38bcce996b15df, e862e38deac4462daf60e1e8d07bc1cc; no voters:
I20260810 15:18:19.505468 21413 raft_consensus.cc:2781] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:19.505481 21020 raft_consensus.cc:2454] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0ff4be8ac6f4f03be38bcce996b15df in term 1.
I20260810 15:18:19.505622 21413 raft_consensus.cc:686] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 1 LEADER]: Becoming Leader. State: Replica: d0ff4be8ac6f4f03be38bcce996b15df, State: Running, Role: LEADER
I20260810 15:18:19.505715 21413 consensus_queue.cc:237] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } }
I20260810 15:18:19.506600 20504 catalog_manager.cc:5495] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df reported cstate change: term changed from 0 to 1, leader changed from <none> to d0ff4be8ac6f4f03be38bcce996b15df (127.29.230.1). New cstate: current_term: 1 leader_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0ff4be8ac6f4f03be38bcce996b15df" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 45051 } health_report { overall_health: HEALTHY } } }
I20260810 15:18:19.517714 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.515286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57172 (local address 127.29.230.1:45051)
0810 15:18:19.515328 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:19.515330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:19.515336 (+ 6us) server_negotiation.cc:407] Connection header received
0810 15:18:19.515364 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:19.515365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:19.515379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:19.515436 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:18:19.515588 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.516026 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.517050 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:19.517249 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:19.517294 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:19.517442 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:18:19.517536 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:18:19.517587 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:19.517642 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:18:19.517657 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:18:19.519654 21261 raft_consensus.cc:1261] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 1 FOLLOWER]: Refusing update from remote peer d0ff4be8ac6f4f03be38bcce996b15df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:19.519661 21020 raft_consensus.cc:1261] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 1 FOLLOWER]: Refusing update from remote peer d0ff4be8ac6f4f03be38bcce996b15df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:19.519966 21434 consensus_queue.cc:1035] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e862e38deac4462daf60e1e8d07bc1cc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:19.520109 21413 consensus_queue.cc:1035] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27818374448a4e968eb48de5548cf6db" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:19.522541 21437 mvcc.cc:205] Tried to move back new op lower bound from 7316992407629795328 to 7316992407575928832. Current Snapshot: MvccSnapshot[applied={T|T < 7316992407629795328}]
I20260810 15:18:19.522688 21439 mvcc.cc:205] Tried to move back new op lower bound from 7316992407629795328 to 7316992407575928832. Current Snapshot: MvccSnapshot[applied={T|T < 7316992407629795328}]
I20260810 15:18:19.523332 21443 mvcc.cc:205] Tried to move back new op lower bound from 7316992407629795328 to 7316992407575928832. Current Snapshot: MvccSnapshot[applied={T|T < 7316992407629795328}]
I20260810 15:18:19.525933 20504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:19.526026 20504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:19.526737 20504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2aede64d27a441db9f2702044f670428: Sending DeleteTablet for 3 replicas of tablet 7477ab2343c745b2aa0b25910cfbc146
I20260810 15:18:19.527189 20952 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:19 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:52954
I20260810 15:18:19.527472 21454 tablet_replica.cc:330] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: stopping tablet replica
I20260810 15:18:19.527475 20738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:19 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:57168
I20260810 15:18:19.527679 21454 raft_consensus.cc:2229] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:19.527690 21227 tablet_service.cc:1514] Processing DeleteTablet for tablet 7477ab2343c745b2aa0b25910cfbc146 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:19 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:56704
I20260810 15:18:19.527865 21454 raft_consensus.cc:2258] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:19.528030 21459 tablet_replica.cc:330] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df: stopping tablet replica
I20260810 15:18:19.528124 21459 raft_consensus.cc:2229] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:18:19.528229 21459 raft_consensus.cc:2258] T 7477ab2343c745b2aa0b25910cfbc146 P d0ff4be8ac6f4f03be38bcce996b15df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:19.528280 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 20573
I20260810 15:18:19.553359 21454 ts_tablet_manager.cc:1905] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:19.528363 21460 tablet_replica.cc:330] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: stopping tablet replica
I20260810 15:18:19.554432 21460 raft_consensus.cc:2229] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:19.554621 21460 raft_consensus.cc:2258] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:19.555195 21460 ts_tablet_manager.cc:1905] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:19.555462 21454 ts_tablet_manager.cc:1918] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:19.555543 21454 log.cc:1196] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/7477ab2343c745b2aa0b25910cfbc146
I20260810 15:18:19.555788 21454 ts_tablet_manager.cc:1939] T 7477ab2343c745b2aa0b25910cfbc146 P 27818374448a4e968eb48de5548cf6db: Deleting consensus metadata
I20260810 15:18:19.556290 20483 catalog_manager.cc:4841] TS 27818374448a4e968eb48de5548cf6db (127.29.230.2:44825): tablet 7477ab2343c745b2aa0b25910cfbc146 (table test-table [id=7573ae0ac8e34927a9e8a3e16e86c213]) successfully deleted
I20260810 15:18:19.557152 21460 ts_tablet_manager.cc:1918] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:19.557226 21460 log.cc:1196] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/7477ab2343c745b2aa0b25910cfbc146
I20260810 15:18:19.557469 21460 ts_tablet_manager.cc:1939] T 7477ab2343c745b2aa0b25910cfbc146 P e862e38deac4462daf60e1e8d07bc1cc: Deleting consensus metadata
I20260810 15:18:19.557911 20484 catalog_manager.cc:4841] TS e862e38deac4462daf60e1e8d07bc1cc (127.29.230.3:41191): tablet 7477ab2343c745b2aa0b25910cfbc146 (table test-table [id=7573ae0ac8e34927a9e8a3e16e86c213]) successfully deleted
W20260810 15:18:19.559226 20484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:45051 (error 108)
I20260810 15:18:19.559530 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 20863
I20260810 15:18:19.580737 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:19.580573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:45051 (local address 127.0.0.1:57182)
0810 15:18:19.580652 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:19.580695 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:45051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 15:18:19.580880 20484 catalog_manager.cc:4584] TS d0ff4be8ac6f4f03be38bcce996b15df (127.29.230.1:45051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7477ab2343c745b2aa0b25910cfbc146: Network error: Client connection negotiation failed: client connection to 127.29.230.1:45051: connect: Connection refused (error 111)
I20260810 15:18:19.588861 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 21100
I20260810 15:18:19.600010 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 20455
I20260810 15:18:19.617024 30616 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 15:18:19.962232 30616 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 15:18:19.962320 30616 mini_ranger.cc:66] Stopping Ranger...
I20260810 15:18:20.306349 30616 mini_ranger.cc:68] Stopped Ranger
2026-08-10 15:18:20.306 UTC [18837] LOG: received smart shutdown request
2026-08-10 15:18:20.311 UTC [18837] LOG: background worker "logical replication launcher" (PID 18848) exited with exit code 1
2026-08-10 15:18:20.311 UTC [18842] LOG: shutting down
2026-08-10 15:18:20.470 UTC [18837] LOG: database system is shut down
2026-08-10T15:18:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48219 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:18:20 dist-test-slave-gjrz krb5kdc[21473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:18:20 dist-test-slave-gjrz krb5kdc[21473](info): set up 2 sockets
Aug 10 15:18:20 dist-test-slave-gjrz krb5kdc[21473](info): commencing operation
krb5kdc: starting...
W20260810 15:18:21.734074 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:18:21 dist-test-slave-gjrz krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:18:21.853586 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42927
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:21.936519 21495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:21.936712 21495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:21.936733 21495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:21.937841 21495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:21.937883 21495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:21.937896 21495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:21.937918 21495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:21.937940 21495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:21.939486 21495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:21.939709 21495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:21.939916 21495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:21.944589 21505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:21.944690 21495 server_base.cc:1034] running on GCE node
W20260810 15:18:21.944597 21500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:21.944599 21501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:21.945080 21495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:21.945366 21495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:21.946529 21495 hybrid_clock.cc:648] HybridClock initialized: now 1786375101946510 us; error 34 us; skew 500 ppm
Aug 10 15:18:21 dist-test-slave-gjrz krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:21.951988 21495 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:21.952489 21495 webserver.cc:460] Webserver started at http://127.29.230.62:39103/ using document root <none> and password file <none>
I20260810 15:18:21.952733 21495 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:21.952785 21495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:21.952913 21495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:21.954411 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "584a68732724488fb5e546ce03a9bfee"
format_stamp: "Formatted at 2026-08-10 15:18:21 on dist-test-slave-gjrz"
server_key: "a83a15ff2628a275f4620067422c9948"
server_key_iv: "b3aa7ac38cf39eb654635e56c99aa00a"
server_key_version: "encryptionkey@0"
I20260810 15:18:21.954831 21495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "584a68732724488fb5e546ce03a9bfee"
format_stamp: "Formatted at 2026-08-10 15:18:21 on dist-test-slave-gjrz"
server_key: "a83a15ff2628a275f4620067422c9948"
server_key_iv: "b3aa7ac38cf39eb654635e56c99aa00a"
server_key_version: "encryptionkey@0"
I20260810 15:18:21.956720 21495 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 15:18:21.957638 21517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:21.957806 21495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:21.957903 21495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "584a68732724488fb5e546ce03a9bfee"
format_stamp: "Formatted at 2026-08-10 15:18:21 on dist-test-slave-gjrz"
server_key: "a83a15ff2628a275f4620067422c9948"
server_key_iv: "b3aa7ac38cf39eb654635e56c99aa00a"
server_key_version: "encryptionkey@0"
I20260810 15:18:21.957978 21495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:21.970494 21495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:21.972499 21495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:21.972635 21495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:21.976648 21495 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:42927
I20260810 15:18:21.976682 21615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:42927 every 8 connection(s)
I20260810 15:18:21.977015 21495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:21.978003 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:21.978631 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 21495
I20260810 15:18:21.978734 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:18:21.978909 30616 external_mini_cluster.cc:1428] Setting key 82103fd50c02885fde482a4d6806b362
I20260810 15:18:21.979784 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:18:21 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.61@KRBTEST.COM" created.
I20260810 15:18:21.989306 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:21.980514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42927 (local address 127.0.0.1:60388)
0810 15:18:21.980740 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:21.980782 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:21.981379 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:21.981657 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:21.981661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:21.981982 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:21.982171 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:21.982176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:21.983206 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:21.983207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:21.983919 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:21.983921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:21.984409 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:21.984410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:21.984475 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:21.984931 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:21.984939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:21.986253 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:21.987861 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:21.987868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:21.987870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:21.988048 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:21.988276 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:21.988277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:21.988278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:21.988356 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:21.988734 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:21.988740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:21.988803 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 15:18:21.988885 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:18:21.989135 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":94,"thread_start_us":98,"threads_started":1}
I20260810 15:18:21.989310 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:21.980522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.29.230.62:42927)
0810 15:18:21.980747 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:21.980757 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:21.980876 (+ 119us) server_negotiation.cc:407] Connection header received
0810 15:18:21.981517 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:21.981519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:21.981535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:21.981619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:21.982270 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:21.983123 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:21.983997 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:21.984316 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:21.984448 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:21.986329 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:21.986336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:21.986337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:21.986344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:21.987776 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:21.988121 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:21.988124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:21.988124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:21.988220 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:21.988411 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:21.988412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:21.988413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:21.988673 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:21.988746 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:21.988919 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 15:18:21.989070 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260810 15:18:21.990123 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:18:21.990432 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:21.990329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41352)
0810 15:18:21.990367 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:21.990410 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 15:18:21.990545 21518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
I20260810 15:18:21.990921 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:21.990843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41364)
0810 15:18:21.990879 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:21.990898 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260810 15:18:21.991014 21617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:40215: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
I20260810 15:18:22.004252 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:40215
--webserver_interface=127.29.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:22.040305 21617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } attempt: 1
I20260810 15:18:22.040843 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.040717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41366)
0810 15:18:22.040775 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.040808 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:18:22.041272 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.041173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41368)
0810 15:18:22.041221 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.041252 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 15:18:22.041373 21617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:40215: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
W20260810 15:18:22.086232 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:22.086433 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:22.086462 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:22.087597 21628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:22.087636 21628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:22.087656 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:22.087671 21628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:22.087695 21628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:22.089257 21628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:40215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:22.089495 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:22.089708 21628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:22.094385 21634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:22.094390 21635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:22.094437 21628 server_base.cc:1034] running on GCE node
W20260810 15:18:22.094372 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:22.094802 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:22.095100 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:22.096266 21628 hybrid_clock.cc:648] HybridClock initialized: now 1786375102096238 us; error 38 us; skew 500 ppm
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:22.098572 21628 init.cc:377] Logged in from keytab as kudu/127.29.230.61@KRBTEST.COM (short username kudu)
I20260810 15:18:22.098995 21628 webserver.cc:460] Webserver started at http://127.29.230.61:38341/ using document root <none> and password file <none>
I20260810 15:18:22.099184 21628 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:22.099226 21628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:22.099357 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:22.100334 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data/instance:
uuid: "65524c0478b8417d98102deabdcad34f"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "f5f674b36581a27169015ea253edf08e"
server_key_iv: "9a8b1e5b0145dd051be67428808a4bb7"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.100656 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal/instance:
uuid: "65524c0478b8417d98102deabdcad34f"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "f5f674b36581a27169015ea253edf08e"
server_key_iv: "9a8b1e5b0145dd051be67428808a4bb7"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.101872 21628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:18:22.102576 21654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:22.102717 21628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:18:22.102792 21628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "65524c0478b8417d98102deabdcad34f"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "f5f674b36581a27169015ea253edf08e"
server_key_iv: "9a8b1e5b0145dd051be67428808a4bb7"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.102854 21628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:22.109512 21617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } attempt: 2
I20260810 15:18:22.109891 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.109763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41374)
0810 15:18:22.109816 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.109856 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:18:22.110189 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.110115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41380)
0810 15:18:22.110151 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.110173 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:18:22.110365 21617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:40215: Network error: Client connection negotiation failed: client connection to 127.29.230.61:40215: connect: Connection refused (error 111)
I20260810 15:18:22.116868 21628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:22.118669 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:22.118805 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:22.122866 21628 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.61:40215
I20260810 15:18:22.122906 21755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.61:40215 every 8 connection(s)
I20260810 15:18:22.123224 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
I20260810 15:18:22.124114 21756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:22.125602 21756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:18:22.129597 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 21628
I20260810 15:18:22.129703 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-1/wal/instance
I20260810 15:18:22.129859 30616 external_mini_cluster.cc:1428] Setting key dfdc5e994fab885b432b748879c7daa4
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:22.134366 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.126259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.29.230.62:42927)
0810 15:18:22.126335 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.126337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.126506 (+ 169us) server_negotiation.cc:407] Connection header received
0810 15:18:22.126947 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.126948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.126963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.127015 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.127646 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.128144 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.129179 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.129380 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.129446 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.131697 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.131699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.131699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.131708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.133104 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.133487 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.133488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.133489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.133552 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.133747 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.133747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.133748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.133876 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.133941 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.134189 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.134280 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:18:22.134404 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.126221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42927 (local address 127.0.0.1:60390)
0810 15:18:22.126373 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.126409 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.126871 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.127073 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.127076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.127365 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.127569 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.127574 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.128243 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.128244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.129104 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.129108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.129465 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.129466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.129574 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.130053 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.130061 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.131593 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.133189 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.133196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.133198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.133406 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.133610 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.133610 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.133610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.133696 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.133932 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.133935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.134039 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.134136 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.134253 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260810 15:18:22.135167 21756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.60@KRBTEST.COM" created.
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:18:22.144385 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.135398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41396)
0810 15:18:22.135438 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.135448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.135486 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.135747 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.135747 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.135870 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.135912 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.135914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.136724 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.136725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.137202 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.137203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.137615 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.137615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.137675 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.138047 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.138051 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.141602 (+ 3551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.143307 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.143309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.143309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.143532 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.143738 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.143738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.143739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.143799 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.144128 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.144130 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.144170 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.144235 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.144319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:18:22.144502 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.135471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.29.230.61:40215)
0810 15:18:22.135624 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.135628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.135646 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:18:22.135662 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.135664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.135673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.135735 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.135974 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.136646 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.137268 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.137553 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.137616 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.141722 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.141732 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.141732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.141739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.143194 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.143627 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.143630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.143630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.143680 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.143854 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.143855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.143855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.144025 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.144135 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.144282 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.144401 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260810 15:18:22.145076 21756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:18:22.145370 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.145267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58128)
0810 15:18:22.145309 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.145352 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:18:22.145490 21660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
I20260810 15:18:22.145872 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.145793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58142)
0810 15:18:22.145828 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.145852 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:18:22.145964 21756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:41037: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
I20260810 15:18:22.156308 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:41037
--webserver_interface=127.29.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:22.195194 21756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } attempt: 1
I20260810 15:18:22.195703 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.195580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58148)
0810 15:18:22.195637 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.195673 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:18:22.196060 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.195969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58158)
0810 15:18:22.196012 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.196039 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:18:22.196157 21756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:41037: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
I20260810 15:18:22.201521 21617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } attempt: 3
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:18:22.208381 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.201849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41398)
0810 15:18:22.201916 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.201927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.201976 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.202095 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.202096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.202257 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.202321 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.202322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.202975 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.202977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.203518 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.203520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.203870 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.203871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.203949 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.204398 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.204402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.205666 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.207385 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.207401 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.207402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.207584 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.207781 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.207782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.207782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.207831 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.208109 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.208111 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.208159 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.208227 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.208313 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 15:18:22.208427 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.201920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.29.230.61:40215)
0810 15:18:22.201972 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.201974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.201985 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:18:22.202028 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.202029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.202040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.202090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.202394 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.202898 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.203593 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.203809 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.203866 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.205747 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.205749 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.205749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.205757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.207303 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.207655 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.207656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.207656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.207722 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.207882 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.207883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.207884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.208021 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.208098 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.208260 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.208342 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:18:22.208775 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:18:22.209093 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.209015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58172)
0810 15:18:22.209045 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.209074 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260810 15:18:22.209389 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.209311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58176)
0810 15:18:22.209349 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.209372 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:18:22.209473 21617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:41037: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
W20260810 15:18:22.239251 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:22.239470 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:22.239499 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:22.240638 21768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:22.240694 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260810 15:18:22.240634 21617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } attempt: 1
W20260810 15:18:22.240713 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:22.240729 21768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:22.240744 21768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:22.241084 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.240942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58188)
0810 15:18:22.241003 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.241050 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:18:22.241411 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.241326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58202)
0810 15:18:22.241363 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.241388 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:18:22.241504 21617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:41037: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
I20260810 15:18:22.242401 21768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42927,127.29.230.61:40215,127.29.230.60:41037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:41037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:22.242646 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:22.242842 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:22.247437 21780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:22.247430 21775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:22.247458 21776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:22.247684 21768 server_base.cc:1034] running on GCE node
I20260810 15:18:22.247911 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:22.248234 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:22.249404 21768 hybrid_clock.cc:648] HybridClock initialized: now 1786375102249389 us; error 27 us; skew 500 ppm
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:22.251906 21768 init.cc:377] Logged in from keytab as kudu/127.29.230.60@KRBTEST.COM (short username kudu)
I20260810 15:18:22.252374 21768 webserver.cc:460] Webserver started at http://127.29.230.60:46743/ using document root <none> and password file <none>
I20260810 15:18:22.252568 21768 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:22.252616 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:22.252735 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:22.253726 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data/instance:
uuid: "b79b619b8a8c491f9acaf74ad6ef1c31"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "9017ecdcb1dece82f670cf480b12c883"
server_key_iv: "02b518b02ada9d8a4b8e0c2e656072fd"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.254109 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal/instance:
uuid: "b79b619b8a8c491f9acaf74ad6ef1c31"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "9017ecdcb1dece82f670cf480b12c883"
server_key_iv: "02b518b02ada9d8a4b8e0c2e656072fd"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.255402 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:18:22.256086 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:22.256237 21768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:18:22.256330 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "b79b619b8a8c491f9acaf74ad6ef1c31"
format_stamp: "Formatted at 2026-08-10 15:18:22 on dist-test-slave-gjrz"
server_key: "9017ecdcb1dece82f670cf480b12c883"
server_key_iv: "02b518b02ada9d8a4b8e0c2e656072fd"
server_key_version: "encryptionkey@0"
I20260810 15:18:22.256399 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:22.264308 21756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } attempt: 2
I20260810 15:18:22.264665 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.264555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58218)
0810 15:18:22.264608 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.264639 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:18:22.264976 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.264888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58232)
0810 15:18:22.264923 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.264950 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:18:22.265060 21756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:41037: Network error: Client connection negotiation failed: client connection to 127.29.230.60:41037: connect: Connection refused (error 111)
I20260810 15:18:22.268673 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:22.270500 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:22.270627 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:22.274447 21768 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.60:41037
I20260810 15:18:22.274485 21895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.60:41037 every 8 connection(s)
I20260810 15:18:22.274783 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
I20260810 15:18:22.275648 21896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:22.277218 21896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:18:22.281457 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 21768
I20260810 15:18:22.281566 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786374947162751-30616-0/minicluster-data/master-2/wal/instance
I20260810 15:18:22.281720 30616 external_mini_cluster.cc:1428] Setting key ba3dc6f69bf4e4a8dc5ae5622138e2a9
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:22.286201 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.277925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.29.230.62:42927)
0810 15:18:22.278016 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.278019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.278142 (+ 123us) server_negotiation.cc:407] Connection header received
0810 15:18:22.278632 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.278634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.278646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.278699 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.279316 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.279784 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.280761 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.280969 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.281027 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.283306 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.283307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.283308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.283315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.284756 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.285274 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.285275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.285275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.285328 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.285569 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.285570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.285570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.285695 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.285761 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.286027 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.286112 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 15:18:22.286330 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.277840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42927 (local address 127.0.0.1:60406)
0810 15:18:22.278020 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.278064 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.278535 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.278776 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.278780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.279022 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.279211 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.279216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.279866 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.279867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.280678 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.280681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.281040 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.281041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.281163 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.281632 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.281654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.283193 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.284853 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.284861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.284863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.285199 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.285398 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.285399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.285399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.285507 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.285755 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.285762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.285890 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.285983 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.286126 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260810 15:18:22.287041 21896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:18:22.290535 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.282086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.29.230.60:41037)
0810 15:18:22.282224 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.282228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.282290 (+ 62us) server_negotiation.cc:407] Connection header received
0810 15:18:22.282359 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.282362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.282382 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.282471 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.283017 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.283877 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.284874 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.285309 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.285357 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.287867 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.287873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.287874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.287879 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.289405 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.289743 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.289746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.289747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.289803 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.290000 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.290001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.290001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.290148 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.290293 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.290364 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.290441 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:18:22.293460 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.287362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.29.230.61:40215)
0810 15:18:22.287425 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.287427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.287435 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:22.287460 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.287461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.287470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.287523 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.287791 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.288266 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.289189 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.289392 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.289451 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.291029 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.291030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.291030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.291039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.292448 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.292770 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.292771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.292772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.292834 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.293042 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.293042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.293042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.293137 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.293181 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.293330 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.293389 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:18:22.293485 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.287285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:40215 (local address 127.0.0.1:41410)
0810 15:18:22.287354 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.287363 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.287401 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.287557 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.287558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.287667 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.287718 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.287719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.288323 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.288324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.289098 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.289101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.289462 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.289463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.289537 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.289848 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.289852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.290949 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.292550 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.292551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.292551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.292706 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.292914 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.292915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.292915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.292981 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.293187 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.293188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.293229 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.293298 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.293384 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 15:18:22.293942 21896 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:18:22.298753 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.291472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.29.230.62:42927)
0810 15:18:22.291526 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.291528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.291536 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:22.291557 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.291558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.291567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.291623 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.292155 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.292755 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.293607 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.293883 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.293914 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.296251 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.296252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.296253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.296260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.297706 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.298020 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.298021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.298021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.298074 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.298255 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.298257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.298257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.298387 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.298575 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.298618 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.298694 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:18:22.300355 21906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.294158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58240)
0810 15:18:22.294391 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.294402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.294461 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.294598 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.294599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.294727 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.294764 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.294765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.295405 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.295406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.296129 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.296131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.296457 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.296458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.296500 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.296787 (+ 287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.296791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.297927 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.299485 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.299487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.299487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.299691 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.299860 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.299861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.299861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.299925 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.300142 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.300142 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.300185 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.300233 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.300302 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260810 15:18:22.300524 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.294198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.29.230.60:41037)
0810 15:18:22.294227 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.294228 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.294449 (+ 221us) server_negotiation.cc:407] Connection header received
0810 15:18:22.294525 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.294525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.294533 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.294572 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.294832 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.295330 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.296200 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.296386 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.296443 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.298019 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.298020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.298020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.298026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.299415 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.299754 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.299755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.299755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.299804 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.299996 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.299997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.299997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.300083 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.300138 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.300268 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.300364 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":13}
I20260810 15:18:22.301919 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: Bootstrap starting.
I20260810 15:18:22.302784 21896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:22.303053 21896 log.cc:824] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: Log is configured to *not* fsync() on all Append() calls
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:18:22.303822 21896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: No bootstrap required, opened a new log
I20260810 15:18:22.305855 21896 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.306012 21896 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:22.306051 21896 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79b619b8a8c491f9acaf74ad6ef1c31, State: Initialized, Role: FOLLOWER
I20260810 15:18:22.306201 21896 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.306316 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.299471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.29.230.61:40215)
0810 15:18:22.299515 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.299516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.299523 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:18:22.299544 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.299544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.299551 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.299589 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.300135 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.300772 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.301402 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.301665 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.301735 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.303752 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.303753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.303753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.303760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.305254 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.305602 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.305603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.305603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.305660 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.305801 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.305802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.305802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.305922 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.306122 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.306166 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.306239 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 15:18:22.306533 21896 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:22.306514 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.306594 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:18:22.308331 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 15:18:22.308772 21950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:18:22.308820 21950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:18:22.309073 21768 master_runner.cc:186] Error getting master registration for 127.29.230.62:42927: OK, instance_id { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" instance_seqno: 1786375101975542 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:18:22.311676 21768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:18:22.316628 21617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } attempt: 2
I20260810 15:18:22.317181 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.310498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41426 (local address 127.29.230.61:40215)
0810 15:18:22.310531 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.310532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.310874 (+ 342us) server_negotiation.cc:407] Connection header received
0810 15:18:22.310978 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.310979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.310989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.311016 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.311761 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.312198 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.312851 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.313045 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.313096 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.314342 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.314343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.314344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.314351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.315762 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.316112 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.316113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.316113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.316168 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.316334 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.316335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.316335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.316436 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.316556 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.317016 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.317095 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:18:22.319435 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.310385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.29.230.62:42927)
0810 15:18:22.310423 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.310425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.310873 (+ 448us) server_negotiation.cc:407] Connection header received
0810 15:18:22.310980 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.310981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.310991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.311018 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.311869 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.312562 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.313572 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.313852 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.313927 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.316516 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.316517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.316518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.316524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.318202 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.318575 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.318576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.318576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.318633 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.318828 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.318829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.318829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.318966 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.319032 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.319238 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.319351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:18:22.324224 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.310691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.29.230.60:41037)
0810 15:18:22.310722 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.310724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.316854 (+ 6130us) server_negotiation.cc:407] Connection header received
0810 15:18:22.316946 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.316947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.316958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.317011 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.318258 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.318950 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.319900 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.320182 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.320217 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.321465 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.321466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.321467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.321474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.323019 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.323375 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.323376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.323376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.323430 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.323629 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.323630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.323630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.323805 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.323896 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.324058 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.324136 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:18:22.325150 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.317076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58256)
0810 15:18:22.317164 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.317172 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.317205 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.317749 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.317750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.317918 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.317960 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.317962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.319338 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.319339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.320061 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.320063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.320729 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.320730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.320789 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.321163 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.321168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.322525 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.324154 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.324155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.324155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.324348 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.324563 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.324564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.324565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.324625 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.324855 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.324856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.324900 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.324960 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.325060 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 15:18:22.325168 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.317285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.29.230.60:41037)
0810 15:18:22.317613 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.317614 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.317622 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:22.317639 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.317640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.317648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.318034 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.318081 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.319224 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.320137 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.320412 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.320920 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.322602 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.322603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.322603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.322610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.324066 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.324426 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.324426 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.324427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.324489 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.324692 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.324693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.324693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.324794 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.324846 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.324990 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.325060 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":89,"threads_started":1}
I20260810 15:18:22.326385 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: Bootstrap starting.
I20260810 15:18:22.327013 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:22.327298 21617 log.cc:824] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:22.328064 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: No bootstrap required, opened a new log
I20260810 15:18:22.329468 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.329579 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:22.329599 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584a68732724488fb5e546ce03a9bfee, State: Initialized, Role: FOLLOWER
I20260810 15:18:22.329705 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.329989 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:22.329970 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.330178 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:18:22.331709 21999 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:18:22.331815 21999 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:18:22.332363 21495 master_runner.cc:186] Error getting master registration for 127.29.230.61:40215: OK, instance_id { permanent_uuid: "65524c0478b8417d98102deabdcad34f" instance_seqno: 1786375102121804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:18:22.335057 21495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:18:22.356195 21756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } attempt: 3
Aug 10 15:18:22 dist-test-slave-gjrz krb5kdc[21473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375102, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:18:22.364310 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.356484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:41037 (local address 127.0.0.1:58268)
0810 15:18:22.356566 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:22.356577 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:22.356613 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:22.356803 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:22.356804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:22.357006 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:22.357060 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.357062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.357950 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.357951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.358782 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:22.358785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.359281 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.359283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:22.359348 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.359860 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:22.359865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:22.361381 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:22.363241 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.363242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.363242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.363484 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.363686 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:22.363686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:22.363687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:22.363752 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:22.363999 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:22.364000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:22.364048 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:18:22.364113 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:18:22.364211 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 15:18:22.364323 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:22.356573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.29.230.60:41037)
0810 15:18:22.356615 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:22.356617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:22.356625 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:22.356703 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:22.356704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:22.356716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:22.356759 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:22.357135 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.357861 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.358854 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:22.359190 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:22.359279 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:22.361464 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:22.361465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:22.361466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:22.361480 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:22.363147 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.363557 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.363558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.363559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.363621 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:22.363807 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:22.363808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:22.363808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:22.363941 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:22.364005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:22.364145 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:18:22.364232 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:18:22.366205 21756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: Bootstrap starting.
I20260810 15:18:22.366966 21756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:22.367283 21756 log.cc:824] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:22.368055 21756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: No bootstrap required, opened a new log
I20260810 15:18:22.369997 21756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.370133 21756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:22.370165 21756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65524c0478b8417d98102deabdcad34f, State: Initialized, Role: FOLLOWER
I20260810 15:18:22.370276 21756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.370580 21756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:22.370543 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.370725 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:18:22.372438 22022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:18:22.372543 22022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:18:22.374348 21628 master_runner.cc:263] No leader master found from master 65524c0478b8417d98102deabdcad34f
I20260810 15:18:22.374400 21628 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:18:22.416968 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:22.417104 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.417340 22003 leader_election.cc:288] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 584a68732724488fb5e546ce03a9bfee (127.29.230.62:42927), b79b619b8a8c491f9acaf74ad6ef1c31 (127.29.230.60:41037)
I20260810 15:18:22.417649 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65524c0478b8417d98102deabdcad34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" is_pre_election: true
I20260810 15:18:22.417810 21848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65524c0478b8417d98102deabdcad34f in term 0.
I20260810 15:18:22.418085 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65524c0478b8417d98102deabdcad34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584a68732724488fb5e546ce03a9bfee" is_pre_election: true
I20260810 15:18:22.418233 21566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65524c0478b8417d98102deabdcad34f in term 0.
I20260810 15:18:22.418427 21660 leader_election.cc:302] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584a68732724488fb5e546ce03a9bfee, 65524c0478b8417d98102deabdcad34f; no voters:
I20260810 15:18:22.418606 22003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:18:22.418663 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:18:22.418692 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:22.419673 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.419799 22003 leader_election.cc:288] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [CANDIDATE]: Term 1 election: Requested vote from peers 584a68732724488fb5e546ce03a9bfee (127.29.230.62:42927), b79b619b8a8c491f9acaf74ad6ef1c31 (127.29.230.60:41037)
I20260810 15:18:22.420042 21848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65524c0478b8417d98102deabdcad34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31"
I20260810 15:18:22.420125 21848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:22.420977 21848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65524c0478b8417d98102deabdcad34f in term 1.
I20260810 15:18:22.421139 21566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65524c0478b8417d98102deabdcad34f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584a68732724488fb5e546ce03a9bfee"
I20260810 15:18:22.421223 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:22.422109 21566 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65524c0478b8417d98102deabdcad34f in term 1.
I20260810 15:18:22.422276 21660 leader_election.cc:302] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584a68732724488fb5e546ce03a9bfee, 65524c0478b8417d98102deabdcad34f; no voters:
I20260810 15:18:22.422405 22003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:22.422595 22003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [term 1 LEADER]: Becoming Leader. State: Replica: 65524c0478b8417d98102deabdcad34f, State: Running, Role: LEADER
I20260810 15:18:22.422694 22003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } }
I20260810 15:18:22.423100 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65524c0478b8417d98102deabdcad34f. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.423183 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:22.423892 22033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:22.424074 22033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:22.425655 21848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [term 1 FOLLOWER]: Refusing update from remote peer 65524c0478b8417d98102deabdcad34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:22.425889 21566 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [term 1 FOLLOWER]: Refusing update from remote peer 65524c0478b8417d98102deabdcad34f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:22.426209 22003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:22.426364 22003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [LEADER]: Connected to new peer: Peer: permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:22.427558 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65524c0478b8417d98102deabdcad34f. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.427647 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:18:22.429572 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65524c0478b8417d98102deabdcad34f. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.429658 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:18:22.430984 22031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.431078 22031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:22.431443 21916 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.431533 21916 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:18:22.431720 22036 mvcc.cc:205] Tried to move back new op lower bound from 7316992419532828672 to 7316992419523887104. Current Snapshot: MvccSnapshot[applied={T|T < 7316992419532828672}]
I20260810 15:18:22.432076 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.432154 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:22.432441 21974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65524c0478b8417d98102deabdcad34f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "584a68732724488fb5e546ce03a9bfee" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42927 } } peers { permanent_uuid: "65524c0478b8417d98102deabdcad34f" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 40215 } } peers { permanent_uuid: "b79b619b8a8c491f9acaf74ad6ef1c31" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 41037 } } }
I20260810 15:18:22.432534 21974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:18:22.433197 22037 mvcc.cc:205] Tried to move back new op lower bound from 7316992419532828672 to 7316992419523887104. Current Snapshot: MvccSnapshot[applied={T|T < 7316992419532828672}]
I20260810 15:18:22.433699 22033 catalog_manager.cc:1347] Generated new cluster ID: 8ca053d16c84434b93e83b44d82fbb22
I20260810 15:18:22.433749 22033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:18:22.440696 22033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:18:22.442325 22033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:22.443656 22039 mvcc.cc:205] Tried to move back new op lower bound from 7316992419532828672 to 7316992419523887104. Current Snapshot: MvccSnapshot[applied={T|T < 7316992419532828672}]
I20260810 15:18:22.462496 22033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65524c0478b8417d98102deabdcad34f: Generated new TSK 0
I20260810 15:18:22.462703 22033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:18:23.309296 21950 catalog_manager.cc:1259] Loaded cluster ID: 8ca053d16c84434b93e83b44d82fbb22
I20260810 15:18:23.309374 21950 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: loading cluster ID for follower catalog manager: success
I20260810 15:18:23.310150 21950 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: acquiring CA information for follower catalog manager: success
I20260810 15:18:23.310448 21950 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b79b619b8a8c491f9acaf74ad6ef1c31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:18:23.332278 21999 catalog_manager.cc:1259] Loaded cluster ID: 8ca053d16c84434b93e83b44d82fbb22
I20260810 15:18:23.332351 21999 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: loading cluster ID for follower catalog manager: success
I20260810 15:18:23.333168 21999 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: acquiring CA information for follower catalog manager: success
I20260810 15:18:23.333467 21999 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 584a68732724488fb5e546ce03a9bfee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:18:23.520022 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 21495
I20260810 15:18:23.533221 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 21628
I20260810 15:18:23.543452 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 21768
2026-08-10T15:18:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2982 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:18:23 dist-test-slave-gjrz krb5kdc[22066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:18:23 dist-test-slave-gjrz krb5kdc[22066](info): set up 2 sockets
Aug 10 15:18:23 dist-test-slave-gjrz krb5kdc[22066](info): commencing operation
krb5kdc: starting...
W20260810 15:18:24.715963 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:18:24 dist-test-slave-gjrz krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:18:24.833350 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34699
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:24.914644 22087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:24.914842 22087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:24.914863 22087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:24.916031 22087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:24.916075 22087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:24.916088 22087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:24.916101 22087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:24.916111 22087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:24.917601 22087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:24.917833 22087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:24.918056 22087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:24.923313 22095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:24.923345 22094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:24.923425 22097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:24.923499 22087 server_base.cc:1034] running on GCE node
I20260810 15:18:24.923743 22087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:24.924047 22087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:24.925184 22087 hybrid_clock.cc:648] HybridClock initialized: now 1786375104925171 us; error 31 us; skew 500 ppm
Aug 10 15:18:24 dist-test-slave-gjrz krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375104, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:24.927628 22087 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:24.928061 22087 webserver.cc:460] Webserver started at http://127.29.230.62:40223/ using document root <none> and password file <none>
I20260810 15:18:24.928285 22087 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:24.928334 22087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:24.928454 22087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:24.929464 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1"
format_stamp: "Formatted at 2026-08-10 15:18:24 on dist-test-slave-gjrz"
server_key: "8c2b5a2b7e042cc43f90984f23714f0a"
server_key_iv: "2eacbdb7a743232c708a16190efd37c8"
server_key_version: "encryptionkey@0"
I20260810 15:18:24.929811 22087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1"
format_stamp: "Formatted at 2026-08-10 15:18:24 on dist-test-slave-gjrz"
server_key: "8c2b5a2b7e042cc43f90984f23714f0a"
server_key_iv: "2eacbdb7a743232c708a16190efd37c8"
server_key_version: "encryptionkey@0"
I20260810 15:18:24.931063 22087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:18:24.931802 22109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:24.931948 22087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:24.932030 22087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1"
format_stamp: "Formatted at 2026-08-10 15:18:24 on dist-test-slave-gjrz"
server_key: "8c2b5a2b7e042cc43f90984f23714f0a"
server_key_iv: "2eacbdb7a743232c708a16190efd37c8"
server_key_version: "encryptionkey@0"
I20260810 15:18:24.932099 22087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:24.954566 22087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:24.956360 22087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:24.956491 22087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:24.960718 22087 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34699
I20260810 15:18:24.960752 22211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34699 every 8 connection(s)
I20260810 15:18:24.961073 22087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:24.961927 22215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:24.964316 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1: Bootstrap starting.
I20260810 15:18:24.964866 22215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:24.965113 22215 log.cc:824] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:24.965874 22215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1: No bootstrap required, opened a new log
I20260810 15:18:24.967618 22215 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } }
I20260810 15:18:24.967764 22215 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:24.967787 22215 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ca1eade8e149d7b6c2fbb9c4982af1, State: Initialized, Role: FOLLOWER
I20260810 15:18:24.967887 22215 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } }
I20260810 15:18:24.967941 22215 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:24.967980 22215 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:24.968030 22215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:24.968346 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 22087
I20260810 15:18:24.968447 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:18:24.968652 30616 external_mini_cluster.cc:1428] Setting key a6017001542e06ee15bab265095b6520
I20260810 15:18:24.968708 22215 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } }
I20260810 15:18:24.968817 22215 leader_election.cc:302] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ca1eade8e149d7b6c2fbb9c4982af1; no voters:
I20260810 15:18:24.968989 22215 leader_election.cc:288] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:18:24.969010 22219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:24.969122 22219 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [term 1 LEADER]: Becoming Leader. State: Replica: b4ca1eade8e149d7b6c2fbb9c4982af1, State: Running, Role: LEADER
I20260810 15:18:24.969226 22219 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } }
I20260810 15:18:24.969269 22215 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:24.969590 22219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } } }
I20260810 15:18:24.969663 22219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:24.969781 22219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ca1eade8e149d7b6c2fbb9c4982af1. Latest consensus state: current_term: 1 leader_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ca1eade8e149d7b6c2fbb9c4982af1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34699 } } }
I20260810 15:18:24.969837 22219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:24.970108 22229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:24.970677 22229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:24.972298 22229 catalog_manager.cc:1347] Generated new cluster ID: 1444297a172b4ffbbf8602bc24debae7
I20260810 15:18:24.972354 22229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:18:24 dist-test-slave-gjrz krb5kdc[22066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:24.980019 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:24.969122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.29.230.62:34699)
0810 15:18:24.969414 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:24.969424 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:24.969453 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:18:24.969548 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:24.969556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:24.969626 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:24.970552 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:24.970692 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:24.972075 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:24.973097 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:24.973739 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:24.973871 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:24.976546 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:24.976550 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:24.976551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:24.976563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:24.978397 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:24.978793 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:24.978796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:24.978797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:24.978915 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:24.979118 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:24.979119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:24.979119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:24.979415 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:24.979567 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:24.979686 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 15:18:24.979814 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260810 15:18:24.980976 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:18:24.987182 22229 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:18:24.987679 22229 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:24.992353 22229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ca1eade8e149d7b6c2fbb9c4982af1: Generated new TSK 0
I20260810 15:18:24.992501 22229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:18:25.034911 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 22087
2026-08-10T15:18:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1496 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:18:25 dist-test-slave-gjrz krb5kdc[22251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:18:25 dist-test-slave-gjrz krb5kdc[22251](info): set up 2 sockets
Aug 10 15:18:25 dist-test-slave-gjrz krb5kdc[22251](info): commencing operation
krb5kdc: starting...
W20260810 15:18:26.225590 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:18:26.342283 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:26.423009 22277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:26.423204 22277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:26.423235 22277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:26.424378 22277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:26.424429 22277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:26.424449 22277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:26.424464 22277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:26.424480 22277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:26.425958 22277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:26.426187 22277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:26.426384 22277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:26.431190 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.431241 22277 server_base.cc:1034] running on GCE node
W20260810 15:18:26.431243 22286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.431203 22291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.431593 22277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:26.431890 22277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:26.433029 22277 hybrid_clock.cc:648] HybridClock initialized: now 1786375106433012 us; error 35 us; skew 500 ppm
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:26.435375 22277 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:26.435745 22277 webserver.cc:460] Webserver started at http://127.29.230.62:38485/ using document root <none> and password file <none>
I20260810 15:18:26.435935 22277 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:26.435971 22277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:26.436055 22277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:26.437073 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "c2ace06b486c4338a8452aef1e2de840"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "b5b888dd3c80c5b849725cbcffab1972"
server_key_iv: "5db6fe16681c752f60d68296e45db58a"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.437366 22277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "c2ace06b486c4338a8452aef1e2de840"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "b5b888dd3c80c5b849725cbcffab1972"
server_key_iv: "5db6fe16681c752f60d68296e45db58a"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.438541 22277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:18:26.439149 22302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.439320 22277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.439393 22277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "c2ace06b486c4338a8452aef1e2de840"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "b5b888dd3c80c5b849725cbcffab1972"
server_key_iv: "5db6fe16681c752f60d68296e45db58a"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.439462 22277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:26.453883 22277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:26.455705 22277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:26.455835 22277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:26.459725 22277 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:40821
I20260810 15:18:26.459842 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:40821 every 8 connection(s)
I20260810 15:18:26.460095 22277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:26.461087 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:26.463514 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap starting.
I20260810 15:18:26.464089 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:26.464349 22399 log.cc:824] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:26.465085 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: No bootstrap required, opened a new log
I20260810 15:18:26.466677 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:26.466830 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:26.466866 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Initialized, Role: FOLLOWER
I20260810 15:18:26.466964 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 22277
I20260810 15:18:26.467029 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:26.467078 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:18:26.467116 22399 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:26.467146 22399 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:26.467185 22399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:26.467314 30616 external_mini_cluster.cc:1428] Setting key 9f92a2f716aaef9263587696d5813358
I20260810 15:18:26.468166 22399 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:26.468269 22399 leader_election.cc:302] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ace06b486c4338a8452aef1e2de840; no voters:
I20260810 15:18:26.468453 22399 leader_election.cc:288] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:18:26.468493 22412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:26.468601 22412 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 LEADER]: Becoming Leader. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Running, Role: LEADER
I20260810 15:18:26.468717 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:26.468760 22412 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:26.469108 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ace06b486c4338a8452aef1e2de840. Latest consensus state: current_term: 1 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:26.469239 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:26.469683 22439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:26.469851 22418 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:26.469934 22418 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:26.470166 22439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:26.471714 22439 catalog_manager.cc:1347] Generated new cluster ID: ddf14b10e3d14ebdaa2fd01645814035
I20260810 15:18:26.471766 22439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:26.478271 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.467826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.29.230.62:40821)
0810 15:18:26.467986 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.467995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.468021 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:18:26.468131 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.468136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.468178 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.468301 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:26.469020 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.470435 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.471327 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.471929 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.472029 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.474711 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:26.474716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:26.474716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:26.474730 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:26.476677 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.477047 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.477050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.477051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.477177 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.477357 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.477358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.477358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.477684 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:26.477850 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.477925 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.478069 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260810 15:18:26.480537 22439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:18:26.481205 22439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:26.485903 22439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Generated new TSK 0
I20260810 15:18:26.486068 22439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:18:26.512185 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:26.592749 22461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:26.592968 22461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:26.592999 22461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:26.594146 22461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:26.594188 22461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:26.594223 22461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:26.595803 22461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:26.596060 22461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:26.596264 22461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:26.601074 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.601034 22467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.601045 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.601258 22461 server_base.cc:1034] running on GCE node
I20260810 15:18:26.601511 22461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:26.601737 22461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:26.602890 22461 hybrid_clock.cc:648] HybridClock initialized: now 1786375106602863 us; error 38 us; skew 500 ppm
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:26.605312 22461 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:26.605727 22461 webserver.cc:460] Webserver started at http://127.29.230.1:40803/ using document root <none> and password file <none>
I20260810 15:18:26.605934 22461 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:26.605983 22461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:26.606108 22461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:26.607115 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "6881d5cce53e4debac121fce6451daef"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "69fda82f0c697a9f1f537bfb3cd078d4"
server_key_iv: "650ff52c27a36b6c7403faafb8ac0f18"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.607462 22461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "6881d5cce53e4debac121fce6451daef"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "69fda82f0c697a9f1f537bfb3cd078d4"
server_key_iv: "650ff52c27a36b6c7403faafb8ac0f18"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.608731 22461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:26.609381 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.609541 22461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:18:26.609627 22461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "6881d5cce53e4debac121fce6451daef"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "69fda82f0c697a9f1f537bfb3cd078d4"
server_key_iv: "650ff52c27a36b6c7403faafb8ac0f18"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.609687 22461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:26.617667 22461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:26.619665 22461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:26.619804 22461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:26.619995 22461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:26.620288 22461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:26.620321 22461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.620347 22461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:26.620362 22461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.625708 22461 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44693
I20260810 15:18:26.625761 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44693 every 8 connection(s)
I20260810 15:18:26.626037 22461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:18:26.627102 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 22461
I20260810 15:18:26.627218 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:18:26.627444 30616 external_mini_cluster.cc:1428] Setting key 43d78205264350b5357951d116fa52fe
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:26.635665 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.626691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:46195 (local address 127.29.230.62:40821)
0810 15:18:26.626739 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.626742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.626921 (+ 179us) server_negotiation.cc:407] Connection header received
0810 15:18:26.627635 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.627637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.627656 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.627720 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:26.628925 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.629600 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.630601 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.630800 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.630857 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.633025 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:26.633026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:26.633026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:26.633033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:26.634265 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.634728 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.634729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.634729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.634776 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.635020 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.635020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.635020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.635116 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:26.635161 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.635527 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.635600 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:18:26.635895 22694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.626667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:46195)
0810 15:18:26.626814 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.626848 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.627483 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.628026 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.628030 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.628640 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:26.628825 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.628830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.629696 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.629697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.630511 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.630514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.630871 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.630872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.630986 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.631439 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:26.631447 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:26.632931 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:26.634347 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.634354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.634356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.634657 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.634830 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.634831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.634831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.634962 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.635170 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:26.635175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:26.635380 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.635513 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.635679 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:18:26.636384 22690 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:26.636472 22690 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:26.636643 22690 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:26.637127 22330 ts_manager.cc:194] Registered new tserver with Master: 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693)
I20260810 15:18:26.637732 22330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:46195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:18:26.654469 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:26.736290 22699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:26.736485 22699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:26.736506 22699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:26.737638 22699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:26.737680 22699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:26.737705 22699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:26.739173 22699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:26.739424 22699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:26.739650 22699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:26.744654 22712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.744801 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.744983 22699 server_base.cc:1034] running on GCE node
W20260810 15:18:26.745067 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.745261 22699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:26.745491 22699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:26.746630 22699 hybrid_clock.cc:648] HybridClock initialized: now 1786375106746621 us; error 27 us; skew 500 ppm
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:26.749081 22699 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:26.749517 22699 webserver.cc:460] Webserver started at http://127.29.230.2:43117/ using document root <none> and password file <none>
I20260810 15:18:26.749742 22699 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:26.749791 22699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:26.749912 22699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:26.750933 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "79512a2819054acfa792ff0f231c6b32"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "21868f85b22e9d5fc45357a241a25ea2"
server_key_iv: "d77b5d0cff749f4c71f338d6d263b952"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.751302 22699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "79512a2819054acfa792ff0f231c6b32"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "21868f85b22e9d5fc45357a241a25ea2"
server_key_iv: "d77b5d0cff749f4c71f338d6d263b952"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.752631 22699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:26.753398 22734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.753584 22699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:26.753677 22699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "79512a2819054acfa792ff0f231c6b32"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "21868f85b22e9d5fc45357a241a25ea2"
server_key_iv: "d77b5d0cff749f4c71f338d6d263b952"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.753747 22699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:26.767377 22699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:26.769721 22699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:26.769867 22699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:26.770113 22699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:26.770435 22699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:26.770469 22699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.770498 22699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:26.770521 22699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.776162 22699 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:33571
I20260810 15:18:26.776227 22936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:33571 every 8 connection(s)
I20260810 15:18:26.776515 22699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:18:26.779776 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 22699
I20260810 15:18:26.779883 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:18:26.780054 30616 external_mini_cluster.cc:1428] Setting key 0baca5af9804b775ee797d886b887488
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:26.785934 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.777220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:56463 (local address 127.29.230.62:40821)
0810 15:18:26.777281 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.777283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.777481 (+ 198us) server_negotiation.cc:407] Connection header received
0810 15:18:26.777951 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.777951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.777960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.778017 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:26.778904 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.779454 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.780625 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.780866 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.780928 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.783115 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:26.783117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:26.783118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:26.783125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:26.784470 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.784895 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.784896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.784897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.784969 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.785278 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.785279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.785279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.785424 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:26.785488 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.785752 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.785845 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:18:26.786100 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.777199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:56463)
0810 15:18:26.777357 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.777399 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.777878 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.778075 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.778078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.778625 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:26.778825 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.778830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.779553 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.779554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.780540 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.780543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.780947 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.780948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.781117 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.781569 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:26.781578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:26.783026 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:26.784575 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.784581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.784583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.784810 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.785061 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.785062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.785062 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.785179 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.785490 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:26.785497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:26.785626 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.785721 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.785874 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260810 15:18:26.786552 22937 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:26.786641 22937 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:26.786806 22937 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:26.787233 22330 ts_manager.cc:194] Registered new tserver with Master: 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571)
I20260810 15:18:26.787756 22330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:56463
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:18:26.805982 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:26.889029 22946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:26.889223 22946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:26.889254 22946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:26.890406 22946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:26.890444 22946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:26.890480 22946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:26.891914 22946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:26.892144 22946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:26.892349 22946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:26.897050 22956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.897046 22958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:26.897029 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:26.897117 22946 server_base.cc:1034] running on GCE node
I20260810 15:18:26.897377 22946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:26.897609 22946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:26.898739 22946 hybrid_clock.cc:648] HybridClock initialized: now 1786375106898731 us; error 26 us; skew 500 ppm
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:26.901218 22946 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:26.901647 22946 webserver.cc:460] Webserver started at http://127.29.230.3:43159/ using document root <none> and password file <none>
I20260810 15:18:26.901859 22946 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:26.901919 22946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:26.902057 22946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:26.903059 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "e51b8c423f1a426a94c188a76b077c05"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "772d76f4c7b73e2f470c56b957da28eb"
server_key_iv: "632a9e7651fd333f9416ce5dc0d1bf72"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.903414 22946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "e51b8c423f1a426a94c188a76b077c05"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "772d76f4c7b73e2f470c56b957da28eb"
server_key_iv: "632a9e7651fd333f9416ce5dc0d1bf72"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.904628 22946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:26.905287 22969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.905433 22946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:18:26.905507 22946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "e51b8c423f1a426a94c188a76b077c05"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "772d76f4c7b73e2f470c56b957da28eb"
server_key_iv: "632a9e7651fd333f9416ce5dc0d1bf72"
server_key_version: "encryptionkey@0"
I20260810 15:18:26.905571 22946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:26.928649 22946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:26.930827 22946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:26.930976 22946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:26.931222 22946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:26.931576 22946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:26.931612 22946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.931643 22946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:26.931659 22946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:26.937184 22946 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36523
I20260810 15:18:26.937242 23176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36523 every 8 connection(s)
I20260810 15:18:26.937553 22946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:26.940940 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 22946
I20260810 15:18:26.941054 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:18:26.941243 30616 external_mini_cluster.cc:1428] Setting key 5d075cdeed9d14056d267c937df002c1
Aug 10 15:18:26 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:26.946527 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.938241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:42575 (local address 127.29.230.62:40821)
0810 15:18:26.938303 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.938305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.938483 (+ 178us) server_negotiation.cc:407] Connection header received
0810 15:18:26.938984 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.938985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.938999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.939053 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:26.940047 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.940537 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.941732 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.942024 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.942094 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.944135 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:26.944136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:26.944137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:26.944143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:26.945376 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.945712 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.945713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.945713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.945759 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.945977 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.945978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.945979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.946102 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:26.946166 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.946397 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.946462 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:18:26.946694 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.938194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:42575)
0810 15:18:26.938368 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.938404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.938885 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.939135 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.939138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.939748 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:26.939943 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.939948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.940631 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.940632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.941664 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.941668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.942108 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.942109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.942253 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.942699 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:26.942707 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:26.944056 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:26.945440 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.945445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.945447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.945652 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.945803 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:26.945803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:26.945803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:26.945892 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:26.946167 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:26.946172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:26.946291 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.946375 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.946493 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260810 15:18:26.947116 23177 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:26.947206 23177 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:26.947405 23177 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:26.947808 22330 ts_manager.cc:194] Registered new tserver with Master: e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
I20260810 15:18:26.948168 22330 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:42575
I20260810 15:18:26.952946 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:18:26.963761 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.954235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.29.230.62:40821)
0810 15:18:26.954292 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.954293 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.954388 (+ 95us) server_negotiation.cc:407] Connection header received
0810 15:18:26.954458 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.954458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.954468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.954509 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:26.955058 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.955604 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.956316 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.956552 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.956608 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.960807 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:26.960809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:26.960809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:26.960817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:26.962165 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.962626 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.962628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.962628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.962708 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:26.962934 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:26.962934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:26.962935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:26.963156 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:26.963253 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.963560 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.963660 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:18:26.965704 22330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:18:26.968096 22330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:18:26.976759 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36523 (local address 127.0.0.1:49212)
0810 15:18:26.972666 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.972674 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.972713 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.973199 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.973200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.973202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:26.973244 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.973245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.974594 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.974595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.975736 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.975739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.976465 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.976466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.976547 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.976562 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.976631 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.976680 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260810 15:18:26.977011 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.29.230.3:36523)
0810 15:18:26.973045 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.973050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.973059 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:26.973093 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.973095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.973109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.973315 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:26.973397 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.974486 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.975815 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.976352 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.976422 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.976487 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.976890 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.976937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":92,"threads_started":1}
I20260810 15:18:26.977552 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44693 (local address 127.0.0.1:50160)
0810 15:18:26.972839 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.972848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.972919 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.973208 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.973210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.973212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:26.973268 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.973270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.974207 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.974208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.975738 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.975739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.977318 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.977319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.977383 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.977395 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.977442 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.977483 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":24,"thread_start_us":36,"threads_started":1}
I20260810 15:18:26.977702 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:33571 (local address 127.0.0.1:55232)
0810 15:18:26.972451 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:26.972495 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:26.972566 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:26.973132 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:26.973135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:26.973137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:26.973216 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.973218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.975829 (+ 2611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.975830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.976487 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:26.976488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.977476 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.977477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:26.977552 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.977556 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:18:26.977594 (+ 38us) client_negotiation.cc:240] Negotiation successful
0810 15:18:26.977636 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":55,"threads_started":1}
I20260810 15:18:26.977746 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.29.230.1:44693)
0810 15:18:26.972849 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.972855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.972909 (+ 54us) server_negotiation.cc:407] Connection header received
0810 15:18:26.973003 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.973005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.973022 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.973114 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:26.973342 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.974118 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.976526 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.977238 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.977314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.977375 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.977573 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.977689 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":101,"threads_started":1}
I20260810 15:18:26.977841 23057 tablet_service.cc:1467] Processing CreateTablet for tablet 331cf8e360124539b3f03f72c63ed715 (DEFAULT_TABLE table=test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:26.978015 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:26.972793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55232 (local address 127.29.230.2:33571)
0810 15:18:26.972949 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:26.972954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:26.972964 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:18:26.973005 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:26.973007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:26.973022 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:26.973289 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:26.973363 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.974486 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.976563 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:26.977174 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:26.977811 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:26.977878 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:26.977905 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 15:18:26.977947 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260810 15:18:26.978240 23057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331cf8e360124539b3f03f72c63ed715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:26.978367 22570 tablet_service.cc:1467] Processing CreateTablet for tablet 331cf8e360124539b3f03f72c63ed715 (DEFAULT_TABLE table=test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:26.978618 22570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331cf8e360124539b3f03f72c63ed715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:26.981104 23194 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Bootstrap starting.
I20260810 15:18:26.981544 23195 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Bootstrap starting.
I20260810 15:18:26.981561 22815 tablet_service.cc:1467] Processing CreateTablet for tablet 331cf8e360124539b3f03f72c63ed715 (DEFAULT_TABLE table=test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:26.981781 23194 tablet_bootstrap.cc:654] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:26.981876 22815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 331cf8e360124539b3f03f72c63ed715. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:26.982077 23195 tablet_bootstrap.cc:654] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:26.982311 23194 log.cc:824] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:26.982499 23195 log.cc:824] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:26.983067 23194 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: No bootstrap required, opened a new log
I20260810 15:18:26.983130 23194 ts_tablet_manager.cc:1397] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:26.983307 23195 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: No bootstrap required, opened a new log
I20260810 15:18:26.983362 23195 ts_tablet_manager.cc:1397] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:26.984654 23200 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Bootstrap starting.
I20260810 15:18:26.985267 23200 tablet_bootstrap.cc:654] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:26.985216 23194 raft_consensus.cc:348] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.985293 23195 raft_consensus.cc:348] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.985380 23194 raft_consensus.cc:374] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:26.985414 23194 raft_consensus.cc:729] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51b8c423f1a426a94c188a76b077c05, State: Initialized, Role: FOLLOWER
I20260810 15:18:26.985411 23195 raft_consensus.cc:374] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:26.985455 23195 raft_consensus.cc:729] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6881d5cce53e4debac121fce6451daef, State: Initialized, Role: FOLLOWER
I20260810 15:18:26.985544 23195 consensus_queue.cc:260] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.985563 23194 consensus_queue.cc:260] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.985795 23200 log.cc:824] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:26.985813 23195 ts_tablet_manager.cc:1428] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:26.985813 23194 ts_tablet_manager.cc:1428] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:18:26.985922 23177 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
I20260810 15:18:26.986027 22690 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
I20260810 15:18:26.986601 23200 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: No bootstrap required, opened a new log
I20260810 15:18:26.986670 23200 ts_tablet_manager.cc:1397] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:26.988164 23200 raft_consensus.cc:348] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.988319 23200 raft_consensus.cc:374] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:26.988358 23200 raft_consensus.cc:729] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79512a2819054acfa792ff0f231c6b32, State: Initialized, Role: FOLLOWER
I20260810 15:18:26.988456 23200 consensus_queue.cc:260] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:26.988686 23200 ts_tablet_manager.cc:1428] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:26.988705 22937 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
W20260810 15:18:27.027060 22940 tablet.cc:2369] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:18:27.126734 22691 tablet.cc:2369] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:18:27.142392 23202 raft_consensus.cc:482] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:27.142524 23202 raft_consensus.cc:504] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:27.142904 23202 leader_election.cc:288] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
I20260810 15:18:27.146927 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.143169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:49957 (local address 127.29.230.2:33571)
0810 15:18:27.143254 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.143257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.143390 (+ 133us) server_negotiation.cc:407] Connection header received
0810 15:18:27.143450 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.143451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.143461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.143509 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.143685 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.144441 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.145974 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.146567 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.146642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.146650 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.146817 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.146856 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 15:18:27.146927 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.143227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:53819 (local address 127.29.230.3:36523)
0810 15:18:27.143290 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.143292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.143345 (+ 53us) server_negotiation.cc:407] Connection header received
0810 15:18:27.143423 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.143424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.143434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.143500 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.143685 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.144462 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.145985 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.146567 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.146641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.146650 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.146824 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.146859 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:18:27.146965 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.143086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36523 (local address 127.29.230.1:53819)
0810 15:18:27.143253 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.143280 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.143356 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.143509 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.143510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.143513 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.143626 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.143627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.144551 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.144552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.145869 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.145873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.146651 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.146652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.146745 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.146752 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.146806 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.146860 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":23,"thread_start_us":82,"threads_started":1}
I20260810 15:18:27.146971 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.143086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:33571 (local address 127.29.230.1:49957)
0810 15:18:27.143320 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.143329 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.143384 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.143516 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.143517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.143518 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.143618 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.143620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.144551 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.144552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.145869 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.145873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.146653 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.146654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.146745 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.146753 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.146806 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.146859 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":111,"thread_start_us":75,"threads_started":1}
I20260810 15:18:27.147346 23095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331cf8e360124539b3f03f72c63ed715" candidate_uuid: "6881d5cce53e4debac121fce6451daef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b8c423f1a426a94c188a76b077c05" is_pre_election: true
I20260810 15:18:27.147452 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331cf8e360124539b3f03f72c63ed715" candidate_uuid: "6881d5cce53e4debac121fce6451daef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79512a2819054acfa792ff0f231c6b32" is_pre_election: true
I20260810 15:18:27.147513 23095 raft_consensus.cc:2454] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6881d5cce53e4debac121fce6451daef in term 0.
I20260810 15:18:27.147614 22863 raft_consensus.cc:2454] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6881d5cce53e4debac121fce6451daef in term 0.
I20260810 15:18:27.147728 22486 leader_election.cc:302] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6881d5cce53e4debac121fce6451daef, e51b8c423f1a426a94c188a76b077c05; no voters:
I20260810 15:18:27.147861 23202 raft_consensus.cc:2781] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:18:27.147912 23202 raft_consensus.cc:482] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:18:27.147928 23202 raft_consensus.cc:3037] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.148631 23202 raft_consensus.cc:504] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:27.148748 23202 leader_election.cc:288] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 1 election: Requested vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
I20260810 15:18:27.148927 22863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331cf8e360124539b3f03f72c63ed715" candidate_uuid: "6881d5cce53e4debac121fce6451daef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79512a2819054acfa792ff0f231c6b32"
I20260810 15:18:27.149001 23095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "331cf8e360124539b3f03f72c63ed715" candidate_uuid: "6881d5cce53e4debac121fce6451daef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b8c423f1a426a94c188a76b077c05"
I20260810 15:18:27.149017 22863 raft_consensus.cc:3037] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.149066 23095 raft_consensus.cc:3037] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.149879 22863 raft_consensus.cc:2454] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6881d5cce53e4debac121fce6451daef in term 1.
I20260810 15:18:27.149879 23095 raft_consensus.cc:2454] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6881d5cce53e4debac121fce6451daef in term 1.
I20260810 15:18:27.150068 22486 leader_election.cc:302] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6881d5cce53e4debac121fce6451daef, e51b8c423f1a426a94c188a76b077c05; no voters:
I20260810 15:18:27.150168 23202 raft_consensus.cc:2781] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:27.150313 23202 raft_consensus.cc:686] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 LEADER]: Becoming Leader. State: Replica: 6881d5cce53e4debac121fce6451daef, State: Running, Role: LEADER
I20260810 15:18:27.150388 23202 consensus_queue.cc:237] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:27.151240 22327 catalog_manager.cc:5495] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef reported cstate change: term changed from 0 to 1, leader changed from <none> to 6881d5cce53e4debac121fce6451daef (127.29.230.1). New cstate: current_term: 1 leader_uuid: "6881d5cce53e4debac121fce6451daef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } health_report { overall_health: UNKNOWN } } }
W20260810 15:18:27.188117 23178 tablet.cc:2369] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:18:27.201349 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.199045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.29.230.1:44693)
0810 15:18:27.199111 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.199113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.199121 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:27.199163 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.199164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.199186 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.199246 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:18:27.199397 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.199881 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.200626 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.200829 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.200884 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.201043 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:18:27.201155 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:18:27.201228 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.201248 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.201273 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 15:18:27.203208 22863 raft_consensus.cc:1261] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Refusing update from remote peer 6881d5cce53e4debac121fce6451daef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:27.203208 23095 raft_consensus.cc:1261] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Refusing update from remote peer 6881d5cce53e4debac121fce6451daef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:27.203583 23220 consensus_queue.cc:1035] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [LEADER]: Connected to new peer: Peer: permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:27.203725 23202 consensus_queue.cc:1035] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:27.206601 23227 mvcc.cc:205] Tried to move back new op lower bound from 7316992439101550592 to 7316992438888611840. Current Snapshot: MvccSnapshot[applied={T|T < 7316992439101550592}]
I20260810 15:18:27.208355 22330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:27.208441 22330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:27.209159 22330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Sending DeleteTablet for 3 replicas of tablet 331cf8e360124539b3f03f72c63ed715
I20260810 15:18:27.209444 23057 tablet_service.cc:1514] Processing DeleteTablet for tablet 331cf8e360124539b3f03f72c63ed715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:49212
I20260810 15:18:27.209514 22570 tablet_service.cc:1514] Processing DeleteTablet for tablet 331cf8e360124539b3f03f72c63ed715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:50160
I20260810 15:18:27.209570 22815 tablet_service.cc:1514] Processing DeleteTablet for tablet 331cf8e360124539b3f03f72c63ed715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:55232
I20260810 15:18:27.209731 23238 tablet_replica.cc:330] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: stopping tablet replica
I20260810 15:18:27.209815 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 22461
I20260810 15:18:27.209829 23238 raft_consensus.cc:2229] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:18:27.209920 23238 raft_consensus.cc:2258] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.209944 23239 tablet_replica.cc:330] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: stopping tablet replica
I20260810 15:18:27.217912 23239 raft_consensus.cc:2229] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:27.218181 23239 raft_consensus.cc:2258] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.221333 23239 ts_tablet_manager.cc:1905] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:27.223476 23239 ts_tablet_manager.cc:1918] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:27.223645 23239 log.cc:1196] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/331cf8e360124539b3f03f72c63ed715
I20260810 15:18:27.224073 23239 ts_tablet_manager.cc:1939] T 331cf8e360124539b3f03f72c63ed715 P 79512a2819054acfa792ff0f231c6b32: Deleting consensus metadata
I20260810 15:18:27.224741 22306 catalog_manager.cc:4841] TS 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): tablet 331cf8e360124539b3f03f72c63ed715 (table test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]) successfully deleted
I20260810 15:18:27.227394 23236 tablet_replica.cc:330] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: stopping tablet replica
I20260810 15:18:27.227514 23236 raft_consensus.cc:2229] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:27.227625 23236 raft_consensus.cc:2258] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.228009 23236 ts_tablet_manager.cc:1905] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:27.229223 23236 ts_tablet_manager.cc:1918] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:27.229282 23236 log.cc:1196] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/331cf8e360124539b3f03f72c63ed715
I20260810 15:18:27.229499 23236 ts_tablet_manager.cc:1939] T 331cf8e360124539b3f03f72c63ed715 P e51b8c423f1a426a94c188a76b077c05: Deleting consensus metadata
I20260810 15:18:27.229878 22305 catalog_manager.cc:4841] TS e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): tablet 331cf8e360124539b3f03f72c63ed715 (table test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]) successfully deleted
W20260810 15:18:27.231314 22306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:44693 (error 108)
I20260810 15:18:27.231601 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 22699
I20260810 15:18:27.234705 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.231673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44693 (local address 127.0.0.1:50180)
0810 15:18:27.234619 (+ 2946us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.234671 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2917}
W20260810 15:18:27.240113 22306 catalog_manager.cc:4584] TS 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 331cf8e360124539b3f03f72c63ed715: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44693: connect: Connection refused (error 111)
I20260810 15:18:27.245433 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 22946
I20260810 15:18:27.257038 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 22277
I20260810 15:18:27.266251 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--webserver_interface=127.29.230.62
--webserver_port=38485
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:27.346118 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:27.346312 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:27.346342 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:27.347466 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:27.347503 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:27.347522 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:27.347537 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:27.347553 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:27.348977 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=38485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:27.349208 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:27.349402 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:27.358632 23256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.358639 23257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.358642 23260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:27.358672 23241 server_base.cc:1034] running on GCE node
I20260810 15:18:27.359110 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:27.359442 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:27.360591 23241 hybrid_clock.cc:648] HybridClock initialized: now 1786375107360579 us; error 30 us; skew 500 ppm
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:27.362964 23241 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:27.363464 23241 webserver.cc:460] Webserver started at http://127.29.230.62:38485/ using document root <none> and password file <none>
I20260810 15:18:27.363684 23241 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:27.363746 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:27.365131 23241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:27.365767 23273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.365983 23241 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:18:27.366086 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "c2ace06b486c4338a8452aef1e2de840"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "b5b888dd3c80c5b849725cbcffab1972"
server_key_iv: "5db6fe16681c752f60d68296e45db58a"
server_key_version: "encryptionkey@0"
I20260810 15:18:27.366508 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:27.384959 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:27.386818 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:27.386950 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:27.390897 23241 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:40821
I20260810 15:18:27.390937 23362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:40821 every 8 connection(s)
I20260810 15:18:27.391319 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:27.393039 23363 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:18:27.393678 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap starting.
I20260810 15:18:27.401360 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 23241
I20260810 15:18:27.401630 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:44693
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=40803
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:27.402051 23363 log.cc:824] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:27.404421 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:18:27.404709 23363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap complete.
I20260810 15:18:27.406858 23363 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:27.407091 23363 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Initialized, Role: FOLLOWER
I20260810 15:18:27.407231 23363 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:27.407320 23363 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:27.407373 23363 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:27.407423 23363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:18:27.408416 23363 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:27.408505 23363 leader_election.cc:302] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ace06b486c4338a8452aef1e2de840; no voters:
I20260810 15:18:27.408656 23363 leader_election.cc:288] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:18:27.408707 23369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:18:27.408828 23369 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 LEADER]: Becoming Leader. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Running, Role: LEADER
I20260810 15:18:27.408921 23363 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:27.408929 23369 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:27.409300 23375 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ace06b486c4338a8452aef1e2de840. Latest consensus state: current_term: 2 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:27.409386 23375 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:27.409597 23377 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:27.409665 23383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:27.409725 23377 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:27.410647 23383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:27.410817 23383 catalog_manager.cc:1259] Loaded cluster ID: ddf14b10e3d14ebdaa2fd01645814035
I20260810 15:18:27.411027 23383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:18:27.411338 23383 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 15:18:27.415993 23383 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 15:18:27.418289 23383 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:27.418488 23383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Loaded TSK: 0
I20260810 15:18:27.418707 23383 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:18:27.484845 23367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:27.485047 23367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:27.485080 23367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:27.486205 23367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:27.486248 23367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:27.486281 23367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:27.487810 23367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:44693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=40803
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:27.488039 23367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:27.488255 23367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:27.494238 23410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.494240 23407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.494598 23406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:27.495853 23367 server_base.cc:1034] running on GCE node
I20260810 15:18:27.496057 23367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:27.496361 23367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:27.497530 23367 hybrid_clock.cc:648] HybridClock initialized: now 1786375107497513 us; error 41 us; skew 500 ppm
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:27.500653 23367 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:27.501207 23367 webserver.cc:460] Webserver started at http://127.29.230.1:40803/ using document root <none> and password file <none>
I20260810 15:18:27.501454 23367 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:27.501530 23367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:27.503293 23367 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:18:27.504035 23426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.504189 23367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:18:27.504285 23367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "6881d5cce53e4debac121fce6451daef"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "69fda82f0c697a9f1f537bfb3cd078d4"
server_key_iv: "650ff52c27a36b6c7403faafb8ac0f18"
server_key_version: "encryptionkey@0"
I20260810 15:18:27.504711 23367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:27.518617 23367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:27.520970 23367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:27.521121 23367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:27.521368 23367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:27.521845 23436 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:18:27.523244 23367 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:18:27.523326 23367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:18:27.523365 23367 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:18:27.524222 23367 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:18:27.524266 23367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:18:27.524329 23436 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Bootstrap starting.
I20260810 15:18:27.531042 23367 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44693
I20260810 15:18:27.531311 23612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44693 every 8 connection(s)
I20260810 15:18:27.531497 23367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:18:27.534066 23436 log.cc:824] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:27.536593 23436 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:18:27.536849 23436 tablet_bootstrap.cc:492] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Bootstrap complete.
I20260810 15:18:27.537645 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 23367
I20260810 15:18:27.537664 23436 ts_tablet_manager.cc:1397] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20260810 15:18:27.537935 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:33571
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=43117
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:27.538905 23436 raft_consensus.cc:348] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:27.539192 23436 raft_consensus.cc:729] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6881d5cce53e4debac121fce6451daef, State: Initialized, Role: FOLLOWER
I20260810 15:18:27.539359 23436 consensus_queue.cc:260] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } }
I20260810 15:18:27.539615 23436 ts_tablet_manager.cc:1428] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:27.545706 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.532213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:58583)
0810 15:18:27.534247 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.534291 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.534886 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.535406 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.535411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.535756 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:27.536011 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.536017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.537700 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.537701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.538960 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.538963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.539845 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.539846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.539981 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.540521 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:27.540530 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:27.541988 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:27.544088 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.544094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.544096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.544352 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.544603 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.544604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.544604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.544715 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.545141 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:27.545147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:27.545285 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.545372 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.545525 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1984,"thread_start_us":99,"threads_started":1}
I20260810 15:18:27.546301 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.532308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:58583 (local address 127.29.230.62:40821)
0810 15:18:27.532482 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.532492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.534418 (+ 1926us) server_negotiation.cc:407] Connection header received
0810 15:18:27.535128 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.535135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.535193 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.535373 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:27.536145 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.537560 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.539061 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.539664 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.539733 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.542082 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:27.542086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:27.542086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:27.542098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:27.543999 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.544433 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.544436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.544437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.544541 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.544774 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.544774 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.544775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.545041 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:27.545105 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.545988 (+ 883us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.546124 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260810 15:18:27.546864 23615 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:27.547004 23615 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:27.547185 23615 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:27.547930 23299 ts_manager.cc:194] Registered new tserver with Master: 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693)
I20260810 15:18:27.548862 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:58583
I20260810 15:18:27.549649 23615 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:18:27.557417 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.548506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44693 (local address 127.0.0.1:50182)
0810 15:18:27.548718 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.548752 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.548815 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.549298 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.549305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.549633 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:27.549743 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.549746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.551051 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.551053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.551732 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.551735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.552392 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.552393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.552470 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.552926 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:27.552930 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:27.554569 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:27.556348 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.556354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.556357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.556550 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.556741 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.556742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.556743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.556802 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.557058 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:27.557064 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:27.557117 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.557192 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.557318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260810 15:18:27.557417 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.548571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.29.230.1:44693)
0810 15:18:27.549145 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.549150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.549160 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:18:27.549191 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.549193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.549207 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.549279 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:27.549838 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.550944 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.551807 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.552264 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.552333 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.554660 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:27.554670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:27.554670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:27.554677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:27.556265 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.556627 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.556630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.556630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.556683 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.556852 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.556853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.556853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.556994 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:27.557070 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.557234 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.557320 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":541,"thread_start_us":79,"threads_started":1}
I20260810 15:18:27.557817 23510 tablet_service.cc:1514] Processing DeleteTablet for tablet 331cf8e360124539b3f03f72c63ed715 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260810 15:18:27.558322 23633 tablet_replica.cc:330] stopping tablet replica
I20260810 15:18:27.558436 23633 raft_consensus.cc:2229] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:27.558506 23633 raft_consensus.cc:2258] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.558817 23633 ts_tablet_manager.cc:1905] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:27.561084 23633 ts_tablet_manager.cc:1918] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:27.561164 23633 log.cc:1196] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/331cf8e360124539b3f03f72c63ed715
I20260810 15:18:27.561390 23633 ts_tablet_manager.cc:1939] T 331cf8e360124539b3f03f72c63ed715 P 6881d5cce53e4debac121fce6451daef: Deleting consensus metadata
I20260810 15:18:27.561923 23280 catalog_manager.cc:4841] TS 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693): tablet 331cf8e360124539b3f03f72c63ed715 (table test-table [id=933aebe0b3e14b4ca9ded8ccb1b4a15f]) successfully deleted
W20260810 15:18:27.625509 23627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:27.625707 23627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:27.625728 23627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:27.626901 23627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:27.626945 23627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:27.626971 23627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:27.628444 23627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:33571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=43117
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:27.628669 23627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:27.628883 23627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:27.633664 23648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.633636 23646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.633646 23645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:27.633919 23627 server_base.cc:1034] running on GCE node
I20260810 15:18:27.634148 23627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:27.634369 23627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:27.635532 23627 hybrid_clock.cc:648] HybridClock initialized: now 1786375107635514 us; error 29 us; skew 500 ppm
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:27.638075 23627 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:27.638485 23627 webserver.cc:460] Webserver started at http://127.29.230.2:43117/ using document root <none> and password file <none>
I20260810 15:18:27.638702 23627 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:27.638767 23627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:27.640101 23627 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:27.640630 23665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.640779 23627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.640859 23627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "79512a2819054acfa792ff0f231c6b32"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "21868f85b22e9d5fc45357a241a25ea2"
server_key_iv: "d77b5d0cff749f4c71f338d6d263b952"
server_key_version: "encryptionkey@0"
I20260810 15:18:27.641170 23627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:27.649794 23627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:27.651757 23627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:27.651885 23627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:27.652079 23627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:27.652352 23627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:27.652385 23627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.652411 23627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:27.652436 23627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.657878 23627 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:33571
I20260810 15:18:27.657917 23873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:33571 every 8 connection(s)
I20260810 15:18:27.658224 23627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:27.665324 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 23627
I20260810 15:18:27.665625 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:36523
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=43159
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:27.667634 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.658893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:43667 (local address 127.29.230.62:40821)
0810 15:18:27.658943 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.658946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.659182 (+ 236us) server_negotiation.cc:407] Connection header received
0810 15:18:27.659711 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.659712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.659725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.659776 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:27.660681 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.661200 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.662199 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.662408 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.662476 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.664736 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:27.664737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:27.664737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:27.664744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:27.666105 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.666486 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.666487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.666487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.666596 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.666888 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.666889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.666889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.667026 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:27.667094 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.667416 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.667537 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:18:27.667749 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.658883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:43667)
0810 15:18:27.659070 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.659104 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.659618 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.659838 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.659841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.660393 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:27.660585 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.660590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.661281 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.661282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.662091 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.662093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.662497 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.662498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.662622 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.663114 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:27.663122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:27.664653 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:27.666184 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.666190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.666192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.666410 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.666690 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.666691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.666692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.666818 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.667108 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:27.667114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:27.667235 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.667378 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.667539 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260810 15:18:27.668372 23874 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:27.668486 23874 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:27.668674 23874 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:27.669051 23299 ts_manager.cc:194] Registered new tserver with Master: 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571)
I20260810 15:18:27.669430 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:43667
W20260810 15:18:27.748420 23878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:27.748626 23878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:27.748649 23878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:27.749778 23878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:27.749814 23878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:27.749843 23878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:27.751310 23878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:36523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=43159
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:27.751559 23878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:27.751766 23878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:27.756676 23888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.756699 23893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:27.756727 23889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:27.756862 23878 server_base.cc:1034] running on GCE node
I20260810 15:18:27.757061 23878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:27.757284 23878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:27.758416 23878 hybrid_clock.cc:648] HybridClock initialized: now 1786375107758404 us; error 29 us; skew 500 ppm
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:27.760766 23878 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:27.761179 23878 webserver.cc:460] Webserver started at http://127.29.230.3:43159/ using document root <none> and password file <none>
I20260810 15:18:27.761386 23878 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:27.761451 23878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:27.762701 23878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:27.763278 23902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.763456 23878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.763538 23878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "e51b8c423f1a426a94c188a76b077c05"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "772d76f4c7b73e2f470c56b957da28eb"
server_key_iv: "632a9e7651fd333f9416ce5dc0d1bf72"
server_key_version: "encryptionkey@0"
I20260810 15:18:27.763860 23878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:27.782440 23878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:27.784372 23878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:27.784511 23878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:27.784718 23878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:27.785006 23878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:27.785046 23878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.785076 23878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:27.785100 23878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.790616 23878 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36523
I20260810 15:18:27.790657 24117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36523 every 8 connection(s)
I20260810 15:18:27.790979 23878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:27.791975 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 23878
Aug 10 15:18:27 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375107, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:27.798676 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.792328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.29.230.62:40821)
0810 15:18:27.792443 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.792445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.792480 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:18:27.792541 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.792541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.792555 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.792611 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:27.793163 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.793892 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.794778 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.795067 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.795141 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.796349 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:27.796350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:27.796350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:27.796357 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:27.797592 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.797911 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.797912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.797912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.797973 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.798143 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.798144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.798144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.798256 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:27.798403 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.798481 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.798560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260810 15:18:27.800624 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.791707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:53359 (local address 127.29.230.62:40821)
0810 15:18:27.791768 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.791769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.791950 (+ 181us) server_negotiation.cc:407] Connection header received
0810 15:18:27.792628 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.792629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.792639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.792688 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:27.793782 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.794442 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.795672 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.795957 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.796043 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.798071 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:27.798072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:27.798072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:27.798078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:27.799413 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.799765 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.799766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.799766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.799821 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:27.800055 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:27.800056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:27.800056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:27.800159 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:27.800212 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.800470 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.800544 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:18:27.800832 24123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.791675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:53359)
0810 15:18:27.791833 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.791875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.792528 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.792758 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.792761 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.793395 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:27.793693 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.793700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.794535 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.794536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.795595 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.795598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.796040 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.796040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.796153 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.796552 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:27.796561 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:27.798003 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:27.799483 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.799490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.799492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.799699 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.799879 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:27.799880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:27.799880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:27.799997 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:27.800215 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:27.800221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:27.800350 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.800435 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.800586 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260810 15:18:27.801237 24120 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40821
I20260810 15:18:27.801323 24120 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:27.801465 24120 heartbeater.cc:511] Master 127.29.230.62:40821 requested a full tablet report, sending...
I20260810 15:18:27.801795 23299 ts_manager.cc:194] Registered new tserver with Master: e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
I20260810 15:18:27.802172 23299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:53359
I20260810 15:18:27.810307 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:18:27.812882 23618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.810662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.29.230.62:40821)
0810 15:18:27.810703 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.810705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.810759 (+ 54us) server_negotiation.cc:407] Connection header received
0810 15:18:27.810821 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.810822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.810851 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.810923 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:18:27.811007 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.811513 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.812235 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.812426 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.812475 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.812639 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:18:27.812736 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:18:27.812808 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.812821 (+ 13us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.812836 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:18:27.813189 23299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:18:27.814764 23299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:18:27.816792 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 (DEFAULT_TABLE table=test-table [id=a333d820ede74e7b86b35b53098bc9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:27.817010 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134af0540c00428fbbf2e3ace6c130d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:27.818305 23436 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Bootstrap starting.
I20260810 15:18:27.818615 23436 tablet_bootstrap.cc:654] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:27.819476 23436 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: No bootstrap required, opened a new log
I20260810 15:18:27.819530 23436 ts_tablet_manager.cc:1397] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:27.819675 23436 raft_consensus.cc:348] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.819736 23436 raft_consensus.cc:374] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:27.819783 23436 raft_consensus.cc:729] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6881d5cce53e4debac121fce6451daef, State: Initialized, Role: FOLLOWER
I20260810 15:18:27.819826 23436 consensus_queue.cc:260] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.819914 23436 ts_tablet_manager.cc:1428] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:27.820241 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.816463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:33571 (local address 127.0.0.1:55242)
0810 15:18:27.816507 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.816518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.816557 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.816960 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.816961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.816962 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.817003 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.817005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.817998 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.817999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.819014 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.819017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.819998 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.819999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.820075 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.820098 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.820130 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.820173 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:18:27.820632 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.816578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.29.230.2:33571)
0810 15:18:27.816761 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.816766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.816776 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:18:27.816819 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.816821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.816836 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.816951 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.817059 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.817914 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.819122 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.819893 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.820401 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.820483 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.820509 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.820554 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260810 15:18:27.821336 23746 tablet_service.cc:1467] Processing CreateTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 (DEFAULT_TABLE table=test-table [id=a333d820ede74e7b86b35b53098bc9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:27.821688 23746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134af0540c00428fbbf2e3ace6c130d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:27.822016 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.816508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36523 (local address 127.0.0.1:49226)
0810 15:18:27.818336 (+ 1828us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.818346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.818399 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.818579 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.818580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.818582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.818624 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.818626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.819760 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.819761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.820926 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.820929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.821792 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.821793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.821873 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.821877 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.821936 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.821968 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1809,"thread_start_us":105,"threads_started":1}
I20260810 15:18:27.822067 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.816734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.29.230.3:36523)
0810 15:18:27.816901 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.816905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.818452 (+ 1547us) server_negotiation.cc:407] Connection header received
0810 15:18:27.818494 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.818496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.818511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.818582 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.818696 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.819650 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.821001 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.821687 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.821761 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.821819 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.821957 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.821998 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260810 15:18:27.822714 23999 tablet_service.cc:1467] Processing CreateTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 (DEFAULT_TABLE table=test-table [id=a333d820ede74e7b86b35b53098bc9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:27.823033 23999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134af0540c00428fbbf2e3ace6c130d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:27.825186 24136 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Bootstrap starting.
I20260810 15:18:27.825763 24136 tablet_bootstrap.cc:654] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:27.826206 24137 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Bootstrap starting.
I20260810 15:18:27.826397 24136 log.cc:824] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:27.826830 24137 tablet_bootstrap.cc:654] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:27.827185 24136 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: No bootstrap required, opened a new log
I20260810 15:18:27.827258 24136 ts_tablet_manager.cc:1397] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:27.827380 24137 log.cc:824] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:27.828114 24137 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: No bootstrap required, opened a new log
I20260810 15:18:27.828186 24137 ts_tablet_manager.cc:1397] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:18:27.829092 24136 raft_consensus.cc:348] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.829258 24136 raft_consensus.cc:374] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:27.829291 24136 raft_consensus.cc:729] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79512a2819054acfa792ff0f231c6b32, State: Initialized, Role: FOLLOWER
I20260810 15:18:27.829404 24136 consensus_queue.cc:260] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.829694 23874 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
I20260810 15:18:27.829685 24136 ts_tablet_manager.cc:1428] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:27.830041 24137 raft_consensus.cc:348] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.830155 24137 raft_consensus.cc:374] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:27.830371 24137 raft_consensus.cc:729] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e51b8c423f1a426a94c188a76b077c05, State: Initialized, Role: FOLLOWER
I20260810 15:18:27.830488 24137 consensus_queue.cc:260] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.830770 24137 ts_tablet_manager.cc:1428] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:18:27.830782 24120 heartbeater.cc:503] Master 127.29.230.62:40821 was elected leader, sending a full tablet report...
I20260810 15:18:27.845760 24140 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:27.845858 24140 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.846160 24140 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523), 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693)
I20260810 15:18:27.848794 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.846370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:40651 (local address 127.29.230.3:36523)
0810 15:18:27.846425 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.846428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.846437 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:27.846482 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.846483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.846491 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.846535 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.846700 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.847163 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848125 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.848476 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848538 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.848547 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.848694 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.848728 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:18:27.848794 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.846288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36523 (local address 127.29.230.2:40651)
0810 15:18:27.846346 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.846358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.846421 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.846555 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.846557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.846558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.846637 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.846639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.847234 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.847235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.848015 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.848018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.848550 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.848635 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.848638 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.848683 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.848724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:18:27.848994 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.846345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44693 (local address 127.29.230.2:59707)
0810 15:18:27.846496 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.846505 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:27.846547 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:27.846694 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:27.846695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:27.846696 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:27.846769 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.846770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.847418 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.847419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.848163 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:27.848165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.848741 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:27.848810 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.848814 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:18:27.848865 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:18:27.848915 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260810 15:18:27.849146 24031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134af0540c00428fbbf2e3ace6c130d6" candidate_uuid: "79512a2819054acfa792ff0f231c6b32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b8c423f1a426a94c188a76b077c05" is_pre_election: true
I20260810 15:18:27.849259 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.846481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:59707 (local address 127.29.230.1:44693)
0810 15:18:27.846532 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.846534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.846543 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:27.846608 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.846609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.846630 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.846680 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:27.846825 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.847337 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848244 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.848660 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.848724 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.848797 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.849171 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.849212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:18:27.849296 24031 raft_consensus.cc:2454] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79512a2819054acfa792ff0f231c6b32 in term 0.
I20260810 15:18:27.849493 23671 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05; no voters:
I20260810 15:18:27.849500 23544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134af0540c00428fbbf2e3ace6c130d6" candidate_uuid: "79512a2819054acfa792ff0f231c6b32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6881d5cce53e4debac121fce6451daef" is_pre_election: true
I20260810 15:18:27.849614 24140 raft_consensus.cc:2781] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:18:27.849647 23544 raft_consensus.cc:2454] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79512a2819054acfa792ff0f231c6b32 in term 0.
I20260810 15:18:27.849673 24140 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:18:27.849692 24140 raft_consensus.cc:3037] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.850401 24140 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.850521 24140 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [CANDIDATE]: Term 1 election: Requested vote from peers e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523), 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693)
I20260810 15:18:27.850672 23544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134af0540c00428fbbf2e3ace6c130d6" candidate_uuid: "79512a2819054acfa792ff0f231c6b32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6881d5cce53e4debac121fce6451daef"
I20260810 15:18:27.850713 24031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134af0540c00428fbbf2e3ace6c130d6" candidate_uuid: "79512a2819054acfa792ff0f231c6b32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e51b8c423f1a426a94c188a76b077c05"
I20260810 15:18:27.850746 23544 raft_consensus.cc:3037] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.850770 24031 raft_consensus.cc:3037] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:27.851481 24031 raft_consensus.cc:2454] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79512a2819054acfa792ff0f231c6b32 in term 1.
I20260810 15:18:27.851485 23544 raft_consensus.cc:2454] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79512a2819054acfa792ff0f231c6b32 in term 1.
I20260810 15:18:27.851651 23672 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6881d5cce53e4debac121fce6451daef, 79512a2819054acfa792ff0f231c6b32; no voters:
I20260810 15:18:27.851790 24140 raft_consensus.cc:2781] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:27.851982 24140 raft_consensus.cc:686] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 1 LEADER]: Becoming Leader. State: Replica: 79512a2819054acfa792ff0f231c6b32, State: Running, Role: LEADER
I20260810 15:18:27.852072 24140 consensus_queue.cc:237] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:27.852821 23299 catalog_manager.cc:5495] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79512a2819054acfa792ff0f231c6b32 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "79512a2819054acfa792ff0f231c6b32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } health_report { overall_health: UNKNOWN } } }
I20260810 15:18:27.867403 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.865103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.29.230.2:33571)
0810 15:18:27.865154 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:27.865156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:27.865163 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:18:27.865199 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:27.865200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:27.865218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:27.865282 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:18:27.865386 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.865928 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.866654 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:27.866855 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:27.866909 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:27.867052 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:18:27.867157 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:18:27.867226 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:27.867289 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 15:18:27.867311 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 15:18:27.869155 24031 raft_consensus.cc:1261] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Refusing update from remote peer 79512a2819054acfa792ff0f231c6b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:27.869189 23544 raft_consensus.cc:1261] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Refusing update from remote peer 79512a2819054acfa792ff0f231c6b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:27.869560 24140 consensus_queue.cc:1035] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:27.869694 24143 consensus_queue.cc:1035] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:27.872200 24151 mvcc.cc:205] Tried to move back new op lower bound from 7316992441829412864 to 7316992441762734080. Current Snapshot: MvccSnapshot[applied={T|T < 7316992441829412864}]
I20260810 15:18:27.873973 23299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:27.874068 23299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:27.874752 23299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Sending DeleteTablet for 3 replicas of tablet 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:27.875033 23510 tablet_service.cc:1514] Processing DeleteTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:50182
I20260810 15:18:27.875110 23746 tablet_service.cc:1514] Processing DeleteTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:55242
I20260810 15:18:27.875142 23633 tablet_replica.cc:330] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: stopping tablet replica
I20260810 15:18:27.875206 23633 raft_consensus.cc:2229] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:27.875176 23999 tablet_service.cc:1514] Processing DeleteTablet for tablet 134af0540c00428fbbf2e3ace6c130d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:27 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:49226
I20260810 15:18:27.875257 23633 raft_consensus.cc:2258] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.875385 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 23367
I20260810 15:18:27.875483 24167 tablet_replica.cc:330] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: stopping tablet replica
I20260810 15:18:27.875586 24167 raft_consensus.cc:2229] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:27.875670 24167 raft_consensus.cc:2258] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.890583 24167 ts_tablet_manager.cc:1905] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:27.892184 24167 ts_tablet_manager.cc:1918] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:27.892246 24167 log.cc:1196] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:27.892454 24167 ts_tablet_manager.cc:1939] T 134af0540c00428fbbf2e3ace6c130d6 P e51b8c423f1a426a94c188a76b077c05: Deleting consensus metadata
I20260810 15:18:27.892864 23276 catalog_manager.cc:4841] TS e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): tablet 134af0540c00428fbbf2e3ace6c130d6 (table test-table [id=a333d820ede74e7b86b35b53098bc9ad]) successfully deleted
I20260810 15:18:27.895447 24164 tablet_replica.cc:330] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: stopping tablet replica
I20260810 15:18:27.895572 24164 raft_consensus.cc:2229] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:18:27.895699 24164 raft_consensus.cc:2258] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:27.896102 24164 ts_tablet_manager.cc:1905] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:27.897389 24164 ts_tablet_manager.cc:1918] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:27.897459 24164 log.cc:1196] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:27.897701 24164 ts_tablet_manager.cc:1939] T 134af0540c00428fbbf2e3ace6c130d6 P 79512a2819054acfa792ff0f231c6b32: Deleting consensus metadata
W20260810 15:18:27.898098 23280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:44693 (error 108)
I20260810 15:18:27.898360 23280 catalog_manager.cc:4841] TS 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): tablet 134af0540c00428fbbf2e3ace6c130d6 (table test-table [id=a333d820ede74e7b86b35b53098bc9ad]) successfully deleted
I20260810 15:18:27.898625 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:27.898503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:44693 (local address 127.0.0.1:50192)
0810 15:18:27.898555 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:27.898598 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:18:27.898614 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 23627
W20260810 15:18:27.898772 23280 catalog_manager.cc:4584] TS 6881d5cce53e4debac121fce6451daef (127.29.230.1:44693): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 134af0540c00428fbbf2e3ace6c130d6: Network error: Client connection negotiation failed: client connection to 127.29.230.1:44693: connect: Connection refused (error 111)
I20260810 15:18:27.909781 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 23878
I20260810 15:18:27.918504 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 23241
I20260810 15:18:27.930147 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--webserver_interface=127.29.230.62
--webserver_port=38485
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:28.011533 24174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:28.011744 24174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:28.011766 24174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:28.012880 24174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:28.012915 24174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:28.012928 24174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:28.012939 24174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:28.012950 24174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:28.014371 24174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=38485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:28.014580 24174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:28.014762 24174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:28.019534 24186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.019543 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.019572 24174 server_base.cc:1034] running on GCE node
W20260810 15:18:28.019510 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.019906 24174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:28.020190 24174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:28.021348 24174 hybrid_clock.cc:648] HybridClock initialized: now 1786375108021325 us; error 39 us; skew 500 ppm
Aug 10 15:18:28 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:28.023769 24174 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:28.024184 24174 webserver.cc:460] Webserver started at http://127.29.230.62:38485/ using document root <none> and password file <none>
I20260810 15:18:28.024398 24174 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:28.024448 24174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:28.025751 24174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:28.026345 24204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.026533 24174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.026623 24174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "c2ace06b486c4338a8452aef1e2de840"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "b5b888dd3c80c5b849725cbcffab1972"
server_key_iv: "5db6fe16681c752f60d68296e45db58a"
server_key_version: "encryptionkey@0"
I20260810 15:18:28.026953 24174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:28.045749 24174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:28.047554 24174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:28.047688 24174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:28.051450 24174 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:40821
I20260810 15:18:28.051494 24299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:40821 every 8 connection(s)
I20260810 15:18:28.051838 24174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:28.053359 24300 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:18:28.054024 24300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap starting.
I20260810 15:18:28.055454 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 24174
I20260810 15:18:28.055706 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:44693
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=40803
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:28.062493 24300 log.cc:824] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:28.064736 24300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:18:28.065023 24300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840: Bootstrap complete.
I20260810 15:18:28.067128 24300 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:28.067394 24300 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Initialized, Role: FOLLOWER
I20260810 15:18:28.067555 24300 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:28.067623 24300 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:28.067664 24300 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:28.067706 24300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:18:28.068737 24300 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:28.068827 24300 leader_election.cc:302] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2ace06b486c4338a8452aef1e2de840; no voters:
I20260810 15:18:28.068984 24300 leader_election.cc:288] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:18:28.069036 24308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:18:28.069161 24308 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [term 3 LEADER]: Becoming Leader. State: Replica: c2ace06b486c4338a8452aef1e2de840, State: Running, Role: LEADER
I20260810 15:18:28.069250 24300 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:28.069253 24308 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } }
I20260810 15:18:28.069698 24312 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:28.069712 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2ace06b486c4338a8452aef1e2de840. Latest consensus state: current_term: 3 leader_uuid: "c2ace06b486c4338a8452aef1e2de840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2ace06b486c4338a8452aef1e2de840" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40821 } } }
I20260810 15:18:28.070065 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:28.070032 24312 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ace06b486c4338a8452aef1e2de840 [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:28.070394 24324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:28.071179 24324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:28.071319 24324 catalog_manager.cc:1259] Loaded cluster ID: ddf14b10e3d14ebdaa2fd01645814035
I20260810 15:18:28.071615 24324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 15:18:28.072034 24324 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1aa03989ad google::LogMessage::Fail() at ??:0
@ 0x7f1aa039b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1aa03984de google::LogMessage::Flush() at ??:0
@ 0x7f1aa0399a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1aa82b90f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1aa82c28b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1aa22660d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1aa225b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1aa30bf6db start_thread at ??:0
@ 0x7f1a9ee9661f clone at ??:0
W20260810 15:18:28.151965 24303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:28.152160 24303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:28.152182 24303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:28.153395 24303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:28.153436 24303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:28.153461 24303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:28.154942 24303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:44693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=40803
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:28.155143 24303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:28.155391 24303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:28.160149 24368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.160153 24373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.160416 24303 server_base.cc:1034] running on GCE node
W20260810 15:18:28.160465 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.160676 24303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:28.160912 24303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:28.162075 24303 hybrid_clock.cc:648] HybridClock initialized: now 1786375108162064 us; error 27 us; skew 500 ppm
Aug 10 15:18:28 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:28.164733 24303 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:28.165182 24303 webserver.cc:460] Webserver started at http://127.29.230.1:40803/ using document root <none> and password file <none>
I20260810 15:18:28.165406 24303 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:28.165452 24303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:28.166715 24303 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:28.167367 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.167547 24303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.167661 24303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "6881d5cce53e4debac121fce6451daef"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "69fda82f0c697a9f1f537bfb3cd078d4"
server_key_iv: "650ff52c27a36b6c7403faafb8ac0f18"
server_key_version: "encryptionkey@0"
I20260810 15:18:28.168115 24303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:28.183918 24303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:28.186421 24303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:28.186600 24303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:28.186903 24303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:28.187451 24400 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:18:28.188905 24303 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:18:28.188964 24303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:18:28.189002 24303 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:18:28.189934 24303 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:18:28.189976 24303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:18:28.190102 24400 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Bootstrap starting.
I20260810 15:18:28.197108 24303 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:44693
I20260810 15:18:28.197368 24587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:44693 every 8 connection(s)
I20260810 15:18:28.197561 24303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:18:28.200105 24400 log.cc:824] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:28.201589 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 24303
I20260810 15:18:28.201881 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:33571
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=43117
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:18:28.202469 24400 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:18:28.202749 24400 tablet_bootstrap.cc:492] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Bootstrap complete.
I20260810 15:18:28.203773 24400 ts_tablet_manager.cc:1397] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260810 15:18:28.204770 24400 raft_consensus.cc:348] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:28.205004 24400 raft_consensus.cc:729] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6881d5cce53e4debac121fce6451daef, State: Initialized, Role: FOLLOWER
I20260810 15:18:28.205145 24400 consensus_queue.cc:260] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:28.205389 24400 ts_tablet_manager.cc:1428] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260810 15:18:28.287114 24598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:28.287388 24598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:28.287423 24598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:28.288973 24598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:28.289027 24598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:28.289062 24598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:28.291081 24598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:33571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=43117
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:28.291392 24598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:28.291651 24598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:28.292867 24596 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:18:28.198290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:49165)
0810 15:18:28.198511 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.198569 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:28.199212 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:28.292701 (+ 93489us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
W20260810 15:18:28.293159 24589 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:40821 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:40821: Client connection negotiation failed: client connection to 127.29.230.62:40821: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:18:28.293509 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.293381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:51185)
0810 15:18:28.293443 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.293486 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:18:28.293819 24596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.293741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:39207)
0810 15:18:28.293782 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.293804 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:18:28.293922 24589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:18:28.297224 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.297231 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.297199 24614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.297396 24598 server_base.cc:1034] running on GCE node
I20260810 15:18:28.297655 24598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:28.297900 24598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:28.299050 24598 hybrid_clock.cc:648] HybridClock initialized: now 1786375108299031 us; error 28 us; skew 500 ppm
Aug 10 15:18:28 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:28.301491 24598 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:28.301915 24598 webserver.cc:460] Webserver started at http://127.29.230.2:43117/ using document root <none> and password file <none>
I20260810 15:18:28.302131 24598 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:28.302181 24598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:28.303493 24598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:28.304036 24625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.304188 24598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:18:28.304275 24598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "79512a2819054acfa792ff0f231c6b32"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "21868f85b22e9d5fc45357a241a25ea2"
server_key_iv: "d77b5d0cff749f4c71f338d6d263b952"
server_key_version: "encryptionkey@0"
I20260810 15:18:28.304642 24598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:28.321554 24598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:28.323522 24598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:28.323661 24598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:28.323873 24598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:28.324177 24598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:28.324218 24598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.324245 24598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:28.324270 24598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.330154 24598 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:33571
I20260810 15:18:28.330209 24832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:33571 every 8 connection(s)
I20260810 15:18:28.330529 24598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:18:28.331602 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.331137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:55499)
0810 15:18:28.331351 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.331442 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
W20260810 15:18:28.332005 24833 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:40821 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:40821: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
I20260810 15:18:28.332285 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.332197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:42443)
0810 15:18:28.332241 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.332266 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:18:28.332521 24837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.332448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:36263)
0810 15:18:28.332483 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.332504 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:18:28.332607 24833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:18:28.338658 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 24598
I20260810 15:18:28.338893 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:36523
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:40821
--webserver_port=43159
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:28.420503 24838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:28.420727 24838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:28.420761 24838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:28.421998 24838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:28.422040 24838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:28.422075 24838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:28.423548 24838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:36523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=43159
--tserver_master_addrs=127.29.230.62:40821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:28.423790 24838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:28.423995 24838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:28.428695 24845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.428850 24852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:28.428877 24848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:28.428910 24838 server_base.cc:1034] running on GCE node
I20260810 15:18:28.429121 24838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:28.429335 24838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:28.430492 24838 hybrid_clock.cc:648] HybridClock initialized: now 1786375108430481 us; error 27 us; skew 500 ppm
Aug 10 15:18:28 dist-test-slave-gjrz krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:28.432845 24838 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:28.433259 24838 webserver.cc:460] Webserver started at http://127.29.230.3:43159/ using document root <none> and password file <none>
I20260810 15:18:28.433483 24838 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:28.433535 24838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:28.434772 24838 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:18:28.435299 24869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.435451 24838 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.435534 24838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "e51b8c423f1a426a94c188a76b077c05"
format_stamp: "Formatted at 2026-08-10 15:18:26 on dist-test-slave-gjrz"
server_key: "772d76f4c7b73e2f470c56b957da28eb"
server_key_iv: "632a9e7651fd333f9416ce5dc0d1bf72"
server_key_version: "encryptionkey@0"
I20260810 15:18:28.435850 24838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:28.455530 24838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:28.457451 24838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:28.457604 24838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:28.457827 24838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:28.458127 24838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:28.458163 24838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.458191 24838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:28.458213 24838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:28.463893 24838 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36523
I20260810 15:18:28.463968 25053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36523 every 8 connection(s)
I20260810 15:18:28.464274 24838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:28.465337 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.464929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:49307)
0810 15:18:28.465108 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.465195 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
W20260810 15:18:28.465715 25056 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:40821 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:40821: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
I20260810 15:18:28.465956 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.465874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:57171)
0810 15:18:28.465913 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.465938 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:18:28.466156 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:28.466108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:51495)
0810 15:18:28.466128 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:28.466144 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260810 15:18:28.466238 25056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:18:28.474454 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 24838
I20260810 15:18:29.294564 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.294338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:50511)
0810 15:18:29.294457 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:29.294499 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260810 15:18:29.333082 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.332874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:35445)
0810 15:18:29.332994 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:29.333033 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260810 15:18:29.466759 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.466531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:36289)
0810 15:18:29.466667 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:29.466707 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260810 15:18:29.843303 25069 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:29.843453 25069 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:29.843839 25069 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
Aug 10 15:18:29 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.2@KRBTEST.COM
Aug 10 15:18:29 dist-test-slave-gjrz krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375108, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.3@KRBTEST.COM
I20260810 15:18:29.853965 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.844028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:33571 (local address 127.29.230.1:57751)
0810 15:18:29.844176 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:29.844187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:29.844247 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:29.845033 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:29.845036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:29.845419 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:29.845615 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:29.845622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.846694 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.846695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:29.847525 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:29.847528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.848092 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.848093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:29.848204 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:29.848705 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:29.848713 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:29.850226 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:29.852056 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:29.852066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:29.852339 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:29.852659 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:29.852661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:29.852796 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:29.853345 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:29.853354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:29.853516 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 15:18:29.853650 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 15:18:29.853833 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":77,"thread_start_us":52,"threads_started":1}
I20260810 15:18:29.853981 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.844142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:57751 (local address 127.29.230.2:33571)
0810 15:18:29.844284 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:29.844295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:29.844316 (+ 21us) server_negotiation.cc:407] Connection header received
0810 15:18:29.844781 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:29.844783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:29.844826 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:29.844940 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:29.845731 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.846588 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.847602 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.848005 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.848108 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:29.850328 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:29.850333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:29.850334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:29.850346 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:29.851973 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852438 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:29.852441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:29.852442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:29.852557 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852903 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:29.852904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:29.852904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:29.853248 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:29.853350 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:29.853712 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 15:18:29.853881 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":50,"threads_started":1}
I20260810 15:18:29.854013 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.844124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:50441 (local address 127.29.230.3:36523)
0810 15:18:29.844268 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:29.844280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:29.844401 (+ 121us) server_negotiation.cc:407] Connection header received
0810 15:18:29.844843 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:29.844846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:29.844883 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:29.845035 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:29.845726 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.846588 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.847604 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.848005 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.848112 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:29.850666 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:29.850670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:29.850670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:29.850680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:29.852231 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852666 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:29.852670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:29.852670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:29.852776 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:29.853010 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:29.853011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:29.853011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:29.853290 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:29.853362 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:29.853712 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 15:18:29.853876 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260810 15:18:29.854017 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:29.843965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36523 (local address 127.29.230.1:50441)
0810 15:18:29.844150 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:29.844161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:29.844222 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:29.845035 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:29.845038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:29.845422 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:29.845616 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:29.845622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.846682 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.846683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:29.847526 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:29.847529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:29.848088 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:29.848089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:29.848205 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:29.848754 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:29.848758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:29.850576 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:29.852333 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:29.852334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:29.852565 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:29.852891 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:29.852892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:29.852892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:29.852948 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:29.853366 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:29.853366 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:29.853516 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 15:18:29.853650 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 15:18:29.853886 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":133,"thread_start_us":53,"threads_started":1}
W20260810 15:18:29.855051 24392 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
W20260810 15:18:29.855130 24393 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:29.855159 24393 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6881d5cce53e4debac121fce6451daef; no voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05
I20260810 15:18:29.855249 25069 raft_consensus.cc:2726] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:18:30.295125 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:30.295001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:47455)
0810 15:18:30.295050 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:30.295085 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:18:30.333662 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:30.333436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:52823)
0810 15:18:30.333568 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:30.333611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260810 15:18:30.467401 25082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:30.467196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:51131)
0810 15:18:30.467327 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:30.467364 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260810 15:18:31.295830 25085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:31.295583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:60401)
0810 15:18:31.295720 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:31.295758 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260810 15:18:31.334316 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:31.334124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:33693)
0810 15:18:31.334237 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:31.334275 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260810 15:18:31.468024 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:31.467804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:56965)
0810 15:18:31.467949 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:31.467985 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260810 15:18:31.563822 25088 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:31.563915 25088 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:31.564136 25088 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
W20260810 15:18:31.564577 24393 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
W20260810 15:18:31.564689 24392 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:31.564738 24392 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6881d5cce53e4debac121fce6451daef; no voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05
I20260810 15:18:31.564808 25088 raft_consensus.cc:2726] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:18:32.296518 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:32.296298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:50945)
0810 15:18:32.296431 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:32.296468 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260810 15:18:32.334965 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:32.334770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:43061)
0810 15:18:32.334889 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:32.334927 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260810 15:18:32.468709 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:32.468485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:40145)
0810 15:18:32.468628 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:32.468666 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
I20260810 15:18:33.082991 25099 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:33.083101 25099 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:33.083312 25099 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
W20260810 15:18:33.083669 24393 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
W20260810 15:18:33.083745 24392 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:33.083786 24392 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6881d5cce53e4debac121fce6451daef; no voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05
I20260810 15:18:33.083853 25099 raft_consensus.cc:2726] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:18:33.297224 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:33.296996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:53333)
0810 15:18:33.297138 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:33.297176 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260810 15:18:33.335692 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:33.335434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:41441)
0810 15:18:33.335577 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:33.335650 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260810 15:18:33.469370 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:33.469174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:38757)
0810 15:18:33.469293 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:33.469329 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260810 15:18:34.297924 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:34.297664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:57435)
0810 15:18:34.297831 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:34.297872 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260810 15:18:34.336393 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:34.336181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:45871)
0810 15:18:34.336317 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:34.336354 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260810 15:18:34.470038 25108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:34.469789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:53961)
0810 15:18:34.469933 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:34.469974 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260810 15:18:35.298676 25111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:35.298468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:33403)
0810 15:18:35.298586 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:35.298626 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260810 15:18:35.337044 25114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:35.336821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:51865)
0810 15:18:35.336960 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:35.336999 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260810 15:18:35.470667 25115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:35.470452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:60155)
0810 15:18:35.470594 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:35.470628 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260810 15:18:36.299379 25119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:36.299140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:47403)
0810 15:18:36.299291 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:36.299333 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260810 15:18:36.337658 25120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:36.337465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:43991)
0810 15:18:36.337585 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:36.337617 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260810 15:18:36.471295 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:36.471036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:58021)
0810 15:18:36.471193 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:36.471228 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260810 15:18:37.299970 25123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:37.299753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:37879)
0810 15:18:37.299884 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:37.299922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260810 15:18:37.338270 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:37.338044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:39241)
0810 15:18:37.338186 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:37.338226 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260810 15:18:37.471920 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:37.471707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:52959)
0810 15:18:37.471832 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:37.471870 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260810 15:18:37.530745 25126 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:37.530843 25126 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:37.531011 25126 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
W20260810 15:18:37.531428 24393 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
W20260810 15:18:37.531533 24392 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:37.531602 24392 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6881d5cce53e4debac121fce6451daef; no voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05
I20260810 15:18:37.531682 25126 raft_consensus.cc:2726] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:18:38.300669 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:38.300437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:54235)
0810 15:18:38.300577 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:38.300620 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260810 15:18:38.338948 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:38.338709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:46847)
0810 15:18:38.338842 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:38.338898 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260810 15:18:38.472546 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:38.472335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:48397)
0810 15:18:38.472467 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:38.472503 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260810 15:18:39.301291 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:39.301078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:35511)
0810 15:18:39.301207 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:39.301246 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260810 15:18:39.339669 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:39.339429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:48753)
0810 15:18:39.339588 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:39.339626 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
I20260810 15:18:39.473146 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:39.472942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:53457)
0810 15:18:39.473070 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:39.473102 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260810 15:18:39.690378 25140 raft_consensus.cc:482] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:39.690500 25140 raft_consensus.cc:504] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79512a2819054acfa792ff0f231c6b32" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 33571 } } peers { permanent_uuid: "e51b8c423f1a426a94c188a76b077c05" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36523 } } peers { permanent_uuid: "6881d5cce53e4debac121fce6451daef" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 44693 } }
I20260810 15:18:39.690645 25140 leader_election.cc:288] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571), e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523)
W20260810 15:18:39.691035 24392 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e51b8c423f1a426a94c188a76b077c05 (127.29.230.3:36523): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
W20260810 15:18:39.691133 24393 leader_election.cc:341] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79512a2819054acfa792ff0f231c6b32 (127.29.230.2:33571): Not found: Tablet not found: 134af0540c00428fbbf2e3ace6c130d6
I20260810 15:18:39.691205 24393 leader_election.cc:302] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6881d5cce53e4debac121fce6451daef; no voters: 79512a2819054acfa792ff0f231c6b32, e51b8c423f1a426a94c188a76b077c05
I20260810 15:18:39.691305 25140 raft_consensus.cc:2726] T 134af0540c00428fbbf2e3ace6c130d6 P 6881d5cce53e4debac121fce6451daef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:18:40.301985 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:40.301749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:49023)
0810 15:18:40.301896 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:40.301936 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260810 15:18:40.340268 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:40.340052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:50521)
0810 15:18:40.340191 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:40.340228 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260810 15:18:40.473799 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:40.473575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:58313)
0810 15:18:40.473715 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:40.473752 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260810 15:18:41.302611 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:41.302396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:39223)
0810 15:18:41.302529 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:41.302566 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260810 15:18:41.340911 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:41.340712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:55263)
0810 15:18:41.340832 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:41.340869 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260810 15:18:41.474467 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:41.474228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:49981)
0810 15:18:41.474378 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:41.474420 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260810 15:18:42.303226 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:42.303007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:36721)
0810 15:18:42.303144 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:42.303178 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260810 15:18:42.341509 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:42.341324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:33825)
0810 15:18:42.341436 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:42.341469 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260810 15:18:42.475101 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:42.474892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:33017)
0810 15:18:42.475019 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:42.475055 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260810 15:18:43.303969 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:43.303777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.1:57579)
0810 15:18:43.303890 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:43.303923 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260810 15:18:43.342152 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:43.341952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.2:60411)
0810 15:18:43.342083 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:43.342114 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260810 15:18:43.475719 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:43.475537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40821 (local address 127.29.230.3:48997)
0810 15:18:43.475646 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:43.475677 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:40821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260810 15:18:43.484172 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 24303
I20260810 15:18:43.497838 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 24598
I20260810 15:18:43.511852 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 24838
2026-08-10T15:18:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18476 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:18:43 dist-test-slave-gjrz krb5kdc[25166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:18:43 dist-test-slave-gjrz krb5kdc[25166](info): set up 2 sockets
Aug 10 15:18:43 dist-test-slave-gjrz krb5kdc[25166](info): commencing operation
krb5kdc: starting...
W20260810 15:18:44.699620 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:18:44 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:18:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:18:44.821636 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:44.903479 25185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:44.903681 25185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:44.903712 25185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:44.904824 25185 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:18:44.904865 25185 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:18:44.904882 25185 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:18:44.904898 25185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:18:44.904914 25185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:44.904932 25185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:44.904955 25185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:18:44.904976 25185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:18:44.906491 25185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:44.906729 25185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:44.906942 25185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:44.912058 25194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:44.912156 25193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:44.912088 25200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:44.912353 25185 server_base.cc:1034] running on GCE node
I20260810 15:18:44.912591 25185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:44.912890 25185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:44.914059 25185 hybrid_clock.cc:648] HybridClock initialized: now 1786375124914040 us; error 27 us; skew 500 ppm
Aug 10 15:18:44 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375124, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:44.916623 25185 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:18:44.917073 25185 webserver.cc:460] Webserver started at http://127.29.230.62:33559/ using document root <none> and password file <none>
I20260810 15:18:44.917301 25185 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:44.917349 25185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:44.917477 25185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:44.918577 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "72392e8ea62f4f31acbddad82a9f660d"
format_stamp: "Formatted at 2026-08-10 15:18:44 on dist-test-slave-gjrz"
server_key: "04e838a8718ac5e4a898ef4708a8eb35"
server_key_iv: "b7291b1a1ff1af471b363062a3c05f0a"
server_key_version: "encryptionkey@0"
I20260810 15:18:44.918910 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "72392e8ea62f4f31acbddad82a9f660d"
format_stamp: "Formatted at 2026-08-10 15:18:44 on dist-test-slave-gjrz"
server_key: "04e838a8718ac5e4a898ef4708a8eb35"
server_key_iv: "b7291b1a1ff1af471b363062a3c05f0a"
server_key_version: "encryptionkey@0"
I20260810 15:18:44.920248 25185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:18:44.920929 25215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:44.921090 25185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:18:44.921176 25185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "72392e8ea62f4f31acbddad82a9f660d"
format_stamp: "Formatted at 2026-08-10 15:18:44 on dist-test-slave-gjrz"
server_key: "04e838a8718ac5e4a898ef4708a8eb35"
server_key_iv: "b7291b1a1ff1af471b363062a3c05f0a"
server_key_version: "encryptionkey@0"
I20260810 15:18:44.921236 25185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:44.931306 25185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:44.933265 25185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:44.933396 25185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:44.937282 25185 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34425
I20260810 15:18:44.937321 25298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34425 every 8 connection(s)
I20260810 15:18:44.937640 25185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:18:44.938505 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:44.940830 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap starting.
I20260810 15:18:44.941437 25301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:44.941708 25301 log.cc:824] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:44.942415 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: No bootstrap required, opened a new log
I20260810 15:18:44.944087 25301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:18:44.944228 25301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:44.944252 25301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Initialized, Role: FOLLOWER
I20260810 15:18:44.944378 25301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:18:44.944445 25301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:18:44.944469 25301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:18:44.944527 25301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:44.945044 25301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:18:44.945135 25301 leader_election.cc:302] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72392e8ea62f4f31acbddad82a9f660d; no voters:
I20260810 15:18:44.945293 25301 leader_election.cc:288] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:18:44.945338 25305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:44.945523 25301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:18:44.945534 25305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 LEADER]: Becoming Leader. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Running, Role: LEADER
I20260810 15:18:44.945741 25305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:18:44.946148 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72392e8ea62f4f31acbddad82a9f660d. Latest consensus state: current_term: 1 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:18:44.946174 25306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:18:44.946334 25306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:44.946676 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 25185
I20260810 15:18:44.946677 25319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:18:44.946679 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:18:44.946779 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:18:44.946959 30616 external_mini_cluster.cc:1428] Setting key 2ec212825ba0efce82b2c56d2282c11f
I20260810 15:18:44.947216 25319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:18:44.948863 25319 catalog_manager.cc:1347] Generated new cluster ID: 9c00dfc2b7c14c198c8d0ea729fd1195
I20260810 15:18:44.948912 25319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:18:44 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:44.958910 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:44.947438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.29.230.62:34425)
0810 15:18:44.947630 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:44.947639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:44.948219 (+ 580us) server_negotiation.cc:407] Connection header received
0810 15:18:44.948372 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:44.948379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:44.948447 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:44.948576 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:44.949449 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:44.950909 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:44.952029 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:44.952648 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:44.952739 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:44.955514 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:44.955518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:44.955518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:44.955531 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:44.957391 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:44.957743 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:44.957745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:44.957746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:44.957853 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:44.958022 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:44.958023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:44.958023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:44.958320 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:44.958465 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:44.958534 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:18:44.958683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260810 15:18:44.965085 25319 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:18:44.965660 25319 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:18:44.972048 25319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Generated new TSK 0
I20260810 15:18:44.972229 25319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:18:45.001807 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:45.083333 25345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:45.083536 25345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:45.083559 25345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:45.084685 25345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:45.084728 25345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:45.084753 25345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:18:45.086192 25345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:45.086403 25345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:45.086618 25345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:45.091517 25356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:45.091589 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:45.091589 25345 server_base.cc:1034] running on GCE node
W20260810 15:18:45.091524 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:45.091885 25345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:45.092111 25345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:45.093277 25345 hybrid_clock.cc:648] HybridClock initialized: now 1786375125093261 us; error 34 us; skew 500 ppm
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:45.095834 25345 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:18:45.096230 25345 webserver.cc:460] Webserver started at http://127.29.230.1:40001/ using document root <none> and password file <none>
I20260810 15:18:45.096418 25345 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:45.096467 25345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:45.096588 25345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:45.097587 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "0452ba26ab5a4c7792404f8a3224c71b"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "a4b09acbbb2cc9b79b0d9d414ce4a146"
server_key_iv: "c2924cfb2f69850b7eaaa7ee49923c82"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.097925 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "0452ba26ab5a4c7792404f8a3224c71b"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "a4b09acbbb2cc9b79b0d9d414ce4a146"
server_key_iv: "c2924cfb2f69850b7eaaa7ee49923c82"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.099150 25345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:18:45.099824 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.099968 25345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.100049 25345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "0452ba26ab5a4c7792404f8a3224c71b"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "a4b09acbbb2cc9b79b0d9d414ce4a146"
server_key_iv: "c2924cfb2f69850b7eaaa7ee49923c82"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.100109 25345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:45.121269 25345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:45.123320 25345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:45.123457 25345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:45.123683 25345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:45.124002 25345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:45.124042 25345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.124073 25345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:45.124096 25345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.129580 25345 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:32967
I20260810 15:18:45.129621 25564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:32967 every 8 connection(s)
I20260810 15:18:45.129925 25345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:45.137583 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 25345
I20260810 15:18:45.137696 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:18:45.137881 30616 external_mini_cluster.cc:1428] Setting key 8e9ab0e19106e39db127b76b66ce8b6c
I20260810 15:18:45.138931 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.130651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:38767 (local address 127.29.230.62:34425)
0810 15:18:45.130711 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.130713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.130888 (+ 175us) server_negotiation.cc:407] Connection header received
0810 15:18:45.131415 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.131416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.131440 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.131493 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:45.132494 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.132992 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.133969 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.134174 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.134228 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.136300 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:45.136302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:45.136302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:45.136310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:45.137444 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.137881 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.137883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.137883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.137938 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.138214 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.138215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.138215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.138361 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:45.138425 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.138751 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.138850 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:18:45.139011 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.130601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:38767)
0810 15:18:45.130771 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.130806 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.131312 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.131581 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.131585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.132191 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:45.132383 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.132389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.133075 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.133076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.133880 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.133883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.134244 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.134245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.134364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.134792 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:45.134800 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:45.136205 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:45.137521 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.137528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.137530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.137815 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.137993 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.137993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.137994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.138116 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.138437 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:45.138444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:45.138606 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.138712 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.138841 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260810 15:18:45.139465 25565 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:18:45.139561 25565 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:45.139761 25565 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:18:45.140319 25244 ts_manager.cc:194] Registered new tserver with Master: 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967)
I20260810 15:18:45.140969 25244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:38767
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:18:45.162372 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:45.242389 25574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:45.242595 25574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:45.242619 25574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:45.243765 25574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:45.243804 25574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:45.243830 25574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:18:45.245306 25574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:45.245527 25574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:45.245743 25574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:45.250712 25580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:45.250738 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:45.250780 25581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:45.250906 25574 server_base.cc:1034] running on GCE node
I20260810 15:18:45.251091 25574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:45.251371 25574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:45.252518 25574 hybrid_clock.cc:648] HybridClock initialized: now 1786375125252499 us; error 33 us; skew 500 ppm
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:45.254964 25574 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:18:45.255447 25574 webserver.cc:460] Webserver started at http://127.29.230.2:43843/ using document root <none> and password file <none>
I20260810 15:18:45.255641 25574 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:45.255686 25574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:45.255803 25574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:45.256829 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "a172a71ee42e411582a97fffcbfd3d05"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "d627f92330fd03f0db6706afe032ed15"
server_key_iv: "248158d7cf6422e0740515b4d0ded471"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.257151 25574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "a172a71ee42e411582a97fffcbfd3d05"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "d627f92330fd03f0db6706afe032ed15"
server_key_iv: "248158d7cf6422e0740515b4d0ded471"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.258338 25574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:18:45.258985 25602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.259130 25574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.259207 25574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "a172a71ee42e411582a97fffcbfd3d05"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "d627f92330fd03f0db6706afe032ed15"
server_key_iv: "248158d7cf6422e0740515b4d0ded471"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.259302 25574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:45.266906 25574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:45.268872 25574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:45.268993 25574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:45.269201 25574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:45.269477 25574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:45.269508 25574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.269527 25574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:45.269550 25574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.275116 25574 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42281
I20260810 15:18:45.275151 25807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42281 every 8 connection(s)
I20260810 15:18:45.275494 25574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:18:45.276877 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 25574
I20260810 15:18:45.276976 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:18:45.277153 30616 external_mini_cluster.cc:1428] Setting key fc0dd3091ad729daf14d2c85ca18c73f
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:45.284752 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.276199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:59115 (local address 127.29.230.62:34425)
0810 15:18:45.276291 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.276293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.276429 (+ 136us) server_negotiation.cc:407] Connection header received
0810 15:18:45.276926 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.276927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.276940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.276995 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:45.278048 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.278562 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.279685 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.279889 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.279949 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.282075 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:45.282077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:45.282077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:45.282083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:45.283379 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.283842 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.283843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.283843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.283889 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.284147 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.284148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.284148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.284245 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:45.284288 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.284618 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.284687 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 15:18:45.285009 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.276124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:59115)
0810 15:18:45.276302 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.276347 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.276824 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.277066 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.277069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.277680 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:45.277926 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.277966 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.278648 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.278649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.279600 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.279603 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.279974 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.279974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.280114 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.280574 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:45.280585 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:45.281974 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:45.283460 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.283468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.283470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.283763 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.283943 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.283944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.283944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.284084 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.284299 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:45.284304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:45.284483 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.284607 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.284777 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260810 15:18:45.285493 25808 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:18:45.285597 25808 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:45.285792 25808 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:18:45.286162 25244 ts_manager.cc:194] Registered new tserver with Master: a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:18:45.286511 25244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:59115
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:18:45.305333 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:18:45.386827 25816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:18:45.387030 25816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:18:45.387053 25816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:18:45.388207 25816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:18:45.388252 25816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:18:45.388276 25816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:18:45.389681 25816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:18:45.389892 25816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:18:45.390110 25816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:18:45.395038 25824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:18:45.395021 25833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:45.395079 25816 server_base.cc:1034] running on GCE node
W20260810 15:18:45.395021 25825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:18:45.395429 25816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:18:45.395686 25816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:18:45.396823 25816 hybrid_clock.cc:648] HybridClock initialized: now 1786375125396807 us; error 32 us; skew 500 ppm
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:18:45.399191 25816 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:18:45.399632 25816 webserver.cc:460] Webserver started at http://127.29.230.3:40997/ using document root <none> and password file <none>
I20260810 15:18:45.399852 25816 fs_manager.cc:359] Metadata directory not provided
I20260810 15:18:45.399896 25816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:18:45.400012 25816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:18:45.400997 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "3d6af5c9b53e0ea8cb477845818110d2"
server_key_iv: "1c330f7488c2e3542fc9c86f8b9dd31a"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.401314 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "3d6af5c9b53e0ea8cb477845818110d2"
server_key_iv: "1c330f7488c2e3542fc9c86f8b9dd31a"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.402576 25816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:18:45.403367 25849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.403530 25816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:18:45.403620 25816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "3d6af5c9b53e0ea8cb477845818110d2"
server_key_iv: "1c330f7488c2e3542fc9c86f8b9dd31a"
server_key_version: "encryptionkey@0"
I20260810 15:18:45.403693 25816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:18:45.417647 25816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:18:45.419540 25816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:18:45.419664 25816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:18:45.419878 25816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:18:45.420159 25816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:18:45.420190 25816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.420218 25816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:18:45.420244 25816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:18:45.426014 25816 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:40511
I20260810 15:18:45.426069 26056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:40511 every 8 connection(s)
I20260810 15:18:45.426352 25816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:18:45.430541 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 25816
I20260810 15:18:45.430655 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:18:45.430848 30616 external_mini_cluster.cc:1428] Setting key 1740dfe39f142482e16d526fabab3af8
Aug 10 15:18:45 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375125, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:18:45.435009 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.426968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:56957 (local address 127.29.230.62:34425)
0810 15:18:45.427029 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.427031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.427222 (+ 191us) server_negotiation.cc:407] Connection header received
0810 15:18:45.427765 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.427766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.427778 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.427830 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:45.428750 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.429256 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.430251 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.430494 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.430568 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.432724 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:45.432725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:45.432726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:45.432733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:45.433746 (+ 1013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.434114 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.434115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.434115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.434172 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.434387 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.434388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.434388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.434502 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:45.434561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.434819 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.434902 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:18:45.435141 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.426939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:56957)
0810 15:18:45.427110 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.427145 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.427668 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.427900 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.427904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.428445 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:18:45.428655 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.428661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.429338 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.429339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.430167 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.430170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.430575 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.430576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.430733 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.431213 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:18:45.431223 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:18:45.432635 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:18:45.433818 (+ 1183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.433825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.433827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.434045 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.434227 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:18:45.434227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:18:45.434228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:18:45.434330 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:18:45.434559 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:18:45.434566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:18:45.434700 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.434787 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.434938 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260810 15:18:45.435560 26057 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:18:45.435662 26057 heartbeater.cc:464] Registering TS with master...
I20260810 15:18:45.435829 26057 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:18:45.436146 25244 ts_manager.cc:194] Registered new tserver with Master: 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511)
I20260810 15:18:45.436498 25244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:56957
I20260810 15:18:45.442502 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:18:45.449476 25325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.443753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.29.230.62:34425)
0810 15:18:45.443810 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.443812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.443903 (+ 91us) server_negotiation.cc:407] Connection header received
0810 15:18:45.443965 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.443966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.443975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.444020 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:18:45.444554 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.445031 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.445780 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.446002 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.446054 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.447182 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:18:45.447183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:18:45.447184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:18:45.447190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:18:45.448441 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.448744 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.448745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.448745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.448799 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:18:45.448957 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:18:45.448958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:18:45.448958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:18:45.449101 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:18:45.449190 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.449313 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.449388 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:18:45.450785 25244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:18:45.452535 25244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:18:45.458442 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32967 (local address 127.0.0.1:60656)
0810 15:18:45.454780 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.454823 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.454889 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.455080 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.455083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.455086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:45.455156 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.455158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456505 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.456506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.457477 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.457480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.458147 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.458148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.458233 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.458248 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.458315 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.458368 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
I20260810 15:18:45.458482 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.29.230.1:32967)
0810 15:18:45.454892 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.454896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.454908 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:18:45.454978 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.454980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.454995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.455069 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:45.455222 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456392 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.457551 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.458071 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.458141 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.458201 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.458348 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.458394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260810 15:18:45.458685 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42281 (local address 127.0.0.1:54986)
0810 15:18:45.454989 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.454998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.455037 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.455579 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.455580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.455582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:45.455624 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.455626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456452 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.456453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.457520 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.457523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.458481 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.458482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.458542 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.458545 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.458590 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.458626 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":59,"threads_started":1}
I20260810 15:18:45.458916 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.29.230.2:42281)
0810 15:18:45.455361 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.455365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.455374 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:45.455410 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.455412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.455426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.455570 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:45.455696 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456363 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.457613 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.458337 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.458393 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.458447 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.458797 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.458844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":82,"threads_started":1}
I20260810 15:18:45.459379 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40511 (local address 127.0.0.1:55844)
0810 15:18:45.454790 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.454825 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.454890 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.455456 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.455458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.455460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:45.455514 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.455516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456864 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.456865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.457964 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.457966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.459129 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.459130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.459197 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.459201 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.459256 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.459316 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":80,"threads_started":1}
I20260810 15:18:45.459426 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.454757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.29.230.3:40511)
0810 15:18:45.455205 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.455210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.455220 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:18:45.455259 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.455281 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.455298 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.455373 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:45.455724 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.456759 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.458048 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.458974 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.459040 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.459095 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.459307 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.459351 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":91,"threads_started":1}
I20260810 15:18:45.459549 25454 tablet_service.cc:1467] Processing CreateTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 (DEFAULT_TABLE table=test-table [id=9d00b1f75cbd418484c5e93b29f7781d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:45.459578 25692 tablet_service.cc:1467] Processing CreateTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 (DEFAULT_TABLE table=test-table [id=9d00b1f75cbd418484c5e93b29f7781d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:45.459901 25692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87681121c92c4b40ba2ec0a0c3170db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:45.459908 25454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87681121c92c4b40ba2ec0a0c3170db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:45.460052 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 (DEFAULT_TABLE table=test-table [id=9d00b1f75cbd418484c5e93b29f7781d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:18:45.460338 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87681121c92c4b40ba2ec0a0c3170db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:18:45.462450 26085 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Bootstrap starting.
I20260810 15:18:45.463131 26085 tablet_bootstrap.cc:654] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:45.463313 26086 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap starting.
I20260810 15:18:45.463723 26088 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Bootstrap starting.
I20260810 15:18:45.463769 26085 log.cc:824] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:45.464028 26086 tablet_bootstrap.cc:654] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:45.464335 26088 tablet_bootstrap.cc:654] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Neither blocks nor log segments found. Creating new log.
I20260810 15:18:45.464556 26086 log.cc:824] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:45.464685 26085 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: No bootstrap required, opened a new log
I20260810 15:18:45.464752 26085 ts_tablet_manager.cc:1397] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:18:45.464875 26088 log.cc:824] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:18:45.465405 26086 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: No bootstrap required, opened a new log
I20260810 15:18:45.465477 26086 ts_tablet_manager.cc:1397] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:45.466609 26085 raft_consensus.cc:348] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.466802 26085 raft_consensus.cc:374] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:45.466845 26085 raft_consensus.cc:729] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a172a71ee42e411582a97fffcbfd3d05, State: Initialized, Role: FOLLOWER
I20260810 15:18:45.466958 26085 consensus_queue.cc:260] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.467243 26085 ts_tablet_manager.cc:1428] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:18:45.467249 25808 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
I20260810 15:18:45.467350 26086 raft_consensus.cc:348] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.467491 26086 raft_consensus.cc:374] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:45.467523 26086 raft_consensus.cc:729] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452ba26ab5a4c7792404f8a3224c71b, State: Initialized, Role: FOLLOWER
I20260810 15:18:45.467636 26086 consensus_queue.cc:260] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.467917 26088 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: No bootstrap required, opened a new log
I20260810 15:18:45.467916 25565 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
I20260810 15:18:45.467973 26088 ts_tablet_manager.cc:1397] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:18:45.468113 26086 ts_tablet_manager.cc:1428] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 15:18:45.469529 26088 raft_consensus.cc:348] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.469713 26088 raft_consensus.cc:374] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:18:45.469760 26088 raft_consensus.cc:729] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe738f597584e02924ec6ee7fbd7b6d, State: Initialized, Role: FOLLOWER
I20260810 15:18:45.469870 26088 consensus_queue.cc:260] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.470129 26088 ts_tablet_manager.cc:1428] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:18:45.470160 26057 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
W20260810 15:18:45.526031 25809 tablet.cc:2369] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:18:45.630620 25568 tablet.cc:2369] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:18:45.676867 26060 tablet.cc:2369] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:18:45.822131 26094 raft_consensus.cc:482] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:18:45.822279 26094 raft_consensus.cc:504] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.822682 26094 leader_election.cc:288] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:18:45.825650 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.822992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:44189 (local address 127.29.230.2:42281)
0810 15:18:45.823052 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.823054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.823178 (+ 124us) server_negotiation.cc:407] Connection header received
0810 15:18:45.823223 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.823224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.823234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.823320 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:45.823479 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.823951 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.824995 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.825349 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.825413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.825421 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.825553 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.825586 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:18:45.825677 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.822992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:37645 (local address 127.29.230.1:32967)
0810 15:18:45.823098 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.823099 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.823108 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:18:45.823136 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.823138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.823147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.823216 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:18:45.823402 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.823958 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.825006 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.825361 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.825433 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.825441 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.825568 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.825611 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 15:18:45.825682 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.822873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32967 (local address 127.29.230.3:37645)
0810 15:18:45.822956 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.822969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.823025 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.823239 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.823241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.823243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:45.823334 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.823337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.824032 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.824033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.824924 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.824927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.825436 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.825436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.825509 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.825512 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.825557 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.825596 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":41}
I20260810 15:18:45.825702 26095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.822880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42281 (local address 127.29.230.3:44189)
0810 15:18:45.823111 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.823120 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:18:45.823168 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:18:45.823339 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:18:45.823340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:18:45.823341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:18:45.823418 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.823420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.824029 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.824030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.824911 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:18:45.824914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.825421 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.825422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:18:45.825489 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.825492 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:18:45.825539 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:18:45.825579 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":86,"thread_start_us":106,"threads_started":1}
I20260810 15:18:45.826026 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87681121c92c4b40ba2ec0a0c3170db2" candidate_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452ba26ab5a4c7792404f8a3224c71b" is_pre_election: true
I20260810 15:18:45.826210 25486 raft_consensus.cc:2454] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe738f597584e02924ec6ee7fbd7b6d in term 0.
I20260810 15:18:45.826174 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87681121c92c4b40ba2ec0a0c3170db2" candidate_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a172a71ee42e411582a97fffcbfd3d05" is_pre_election: true
I20260810 15:18:45.826299 25728 raft_consensus.cc:2454] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fe738f597584e02924ec6ee7fbd7b6d in term 0.
I20260810 15:18:45.826436 25859 leader_election.cc:302] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b, 1fe738f597584e02924ec6ee7fbd7b6d; no voters:
I20260810 15:18:45.826570 26094 raft_consensus.cc:2781] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:18:45.826640 26094 raft_consensus.cc:482] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:18:45.826658 26094 raft_consensus.cc:3037] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:45.827417 26094 raft_consensus.cc:504] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.827559 26094 leader_election.cc:288] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [CANDIDATE]: Term 1 election: Requested vote from peers 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:18:45.827744 25728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87681121c92c4b40ba2ec0a0c3170db2" candidate_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a172a71ee42e411582a97fffcbfd3d05"
I20260810 15:18:45.827750 25486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87681121c92c4b40ba2ec0a0c3170db2" candidate_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0452ba26ab5a4c7792404f8a3224c71b"
I20260810 15:18:45.827834 25728 raft_consensus.cc:3037] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:45.827834 25486 raft_consensus.cc:3037] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:18:45.828732 25486 raft_consensus.cc:2454] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe738f597584e02924ec6ee7fbd7b6d in term 1.
I20260810 15:18:45.828732 25728 raft_consensus.cc:2454] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fe738f597584e02924ec6ee7fbd7b6d in term 1.
I20260810 15:18:45.828943 25859 leader_election.cc:302] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b, 1fe738f597584e02924ec6ee7fbd7b6d; no voters:
I20260810 15:18:45.829026 26094 raft_consensus.cc:2781] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:18:45.829219 26094 raft_consensus.cc:686] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 LEADER]: Becoming Leader. State: Replica: 1fe738f597584e02924ec6ee7fbd7b6d, State: Running, Role: LEADER
I20260810 15:18:45.829319 26094 consensus_queue.cc:237] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:18:45.830149 25243 catalog_manager.cc:5495] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3). New cstate: current_term: 1 leader_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } health_report { overall_health: HEALTHY } } }
I20260810 15:18:45.901432 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.899105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.29.230.3:40511)
0810 15:18:45.899162 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:18:45.899164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:18:45.899172 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:18:45.899210 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:18:45.899211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:18:45.899235 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:18:45.899330 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:18:45.899521 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.900050 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.900755 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:18:45.900981 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:18:45.901033 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:18:45.901137 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:18:45.901250 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:18:45.901308 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:18:45.901321 (+ 13us) server_negotiation.cc:299] Negotiation successful
0810 15:18:45.901341 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:18:45.903388 25728 raft_consensus.cc:1261] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Refusing update from remote peer 1fe738f597584e02924ec6ee7fbd7b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:45.903406 25486 raft_consensus.cc:1261] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Refusing update from remote peer 1fe738f597584e02924ec6ee7fbd7b6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:18:45.903770 26094 consensus_queue.cc:1035] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:45.903935 26096 consensus_queue.cc:1035] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:18:45.906344 26107 mvcc.cc:205] Tried to move back new op lower bound from 7316992515696967680 to 7316992515397365760. Current Snapshot: MvccSnapshot[applied={T|T < 7316992515696967680}]
I20260810 15:18:45.906347 26104 mvcc.cc:205] Tried to move back new op lower bound from 7316992515696967680 to 7316992515397365760. Current Snapshot: MvccSnapshot[applied={T|T < 7316992515696967680}]
I20260810 15:18:45.906711 26103 mvcc.cc:205] Tried to move back new op lower bound from 7316992515696967680 to 7316992515397365760. Current Snapshot: MvccSnapshot[applied={T|T < 7316992515696967680}]
I20260810 15:18:45.909075 25244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:45.909168 25244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:18:45.909925 25244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Sending DeleteTablet for 3 replicas of tablet 87681121c92c4b40ba2ec0a0c3170db2
I20260810 15:18:45.910297 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:45 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:55844
I20260810 15:18:45.910290 25692 tablet_service.cc:1514] Processing DeleteTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:45 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260810 15:18:45.910522 26121 tablet_replica.cc:330] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: stopping tablet replica
I20260810 15:18:45.910552 25454 tablet_service.cc:1514] Processing DeleteTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:45 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:60656
I20260810 15:18:45.910629 26121 raft_consensus.cc:2229] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:18:45.910764 26121 raft_consensus.cc:2258] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:45.910768 26133 tablet_replica.cc:330] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: stopping tablet replica
I20260810 15:18:45.910856 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 25345
I20260810 15:18:45.910912 26133 raft_consensus.cc:2229] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:45.913934 26121 ts_tablet_manager.cc:1905] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:45.915844 26121 ts_tablet_manager.cc:1918] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:45.920003 26124 tablet_replica.cc:330] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: stopping tablet replica
I20260810 15:18:45.920166 26124 raft_consensus.cc:2229] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:18:45.920306 26124 raft_consensus.cc:2258] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:18:45.920475 26121 log.cc:1196] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/87681121c92c4b40ba2ec0a0c3170db2
I20260810 15:18:45.920816 26121 ts_tablet_manager.cc:1939] T 87681121c92c4b40ba2ec0a0c3170db2 P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting consensus metadata
I20260810 15:18:45.921334 25223 catalog_manager.cc:4841] TS 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): tablet 87681121c92c4b40ba2ec0a0c3170db2 (table test-table [id=9d00b1f75cbd418484c5e93b29f7781d]) successfully deleted
I20260810 15:18:45.922114 26124 ts_tablet_manager.cc:1905] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:18:45.923588 26124 ts_tablet_manager.cc:1918] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:18:45.923661 26124 log.cc:1196] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/87681121c92c4b40ba2ec0a0c3170db2
I20260810 15:18:45.923893 26124 ts_tablet_manager.cc:1939] T 87681121c92c4b40ba2ec0a0c3170db2 P a172a71ee42e411582a97fffcbfd3d05: Deleting consensus metadata
I20260810 15:18:45.924301 25218 catalog_manager.cc:4841] TS a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): tablet 87681121c92c4b40ba2ec0a0c3170db2 (table test-table [id=9d00b1f75cbd418484c5e93b29f7781d]) successfully deleted
W20260810 15:18:45.926669 25224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:32967 (error 108)
I20260810 15:18:45.926905 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 25574
I20260810 15:18:45.934775 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:18:45.927115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32967 (local address 127.0.0.1:60664)
0810 15:18:45.934667 (+ 7552us) negotiation.cc:107] Waiting for socket to connect
0810 15:18:45.934734 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7519}
W20260810 15:18:45.937430 25224 catalog_manager.cc:4584] TS 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87681121c92c4b40ba2ec0a0c3170db2: Network error: Client connection negotiation failed: client connection to 127.29.230.1:32967: connect: Connection refused (error 111)
I20260810 15:18:45.942895 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 25816
I20260810 15:18:45.947758 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 25185
I20260810 15:19:05.962333 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--webserver_interface=127.29.230.62
--webserver_port=33559
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:06.044811 26136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:06.045022 26136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:06.045045 26136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:06.046182 26136 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:19:06.046221 26136 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:19:06.046233 26136 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:19:06.046243 26136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:06.046257 26136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:06.046277 26136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:06.046300 26136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:06.046322 26136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:06.047876 26136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=33559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:06.048122 26136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:06.048336 26136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:06.057318 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.057327 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.057332 26144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:06.057545 26136 server_base.cc:1034] running on GCE node
I20260810 15:19:06.057916 26136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:06.058213 26136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:06.059324 26136 hybrid_clock.cc:648] HybridClock initialized: now 1786375146059310 us; error 28 us; skew 500 ppm
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:06.065537 26136 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:19:06.065969 26136 webserver.cc:460] Webserver started at http://127.29.230.62:33559/ using document root <none> and password file <none>
I20260810 15:19:06.066187 26136 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:06.066239 26136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:06.067601 26136 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:19:06.068295 26161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.068497 26136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:19:06.068581 26136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "72392e8ea62f4f31acbddad82a9f660d"
format_stamp: "Formatted at 2026-08-10 15:18:44 on dist-test-slave-gjrz"
server_key: "04e838a8718ac5e4a898ef4708a8eb35"
server_key_iv: "b7291b1a1ff1af471b363062a3c05f0a"
server_key_version: "encryptionkey@0"
I20260810 15:19:06.068979 26136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:06.083895 26136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:06.085839 26136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:06.085976 26136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:06.090152 26136 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34425
I20260810 15:19:06.090201 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34425 every 8 connection(s)
I20260810 15:19:06.090570 26136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:06.092103 26258 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:19:06.092754 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap starting.
I20260810 15:19:06.098354 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 26136
I20260810 15:19:06.098676 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:32967
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=40001
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:19:06.101649 26258 log.cc:824] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:06.103710 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:19:06.103982 26258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap complete.
I20260810 15:19:06.106091 26258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:06.106319 26258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Initialized, Role: FOLLOWER
I20260810 15:19:06.106465 26258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:06.106530 26258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:06.106564 26258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:06.106621 26258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 1 FOLLOWER]: Advancing to term 2
I20260810 15:19:06.107682 26258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:06.107775 26258 leader_election.cc:302] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72392e8ea62f4f31acbddad82a9f660d; no voters:
I20260810 15:19:06.107944 26258 leader_election.cc:288] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 15:19:06.107988 26270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Leader election won for term 2
I20260810 15:19:06.108167 26270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 LEADER]: Becoming Leader. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Running, Role: LEADER
I20260810 15:19:06.108222 26258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:06.108260 26270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:06.108564 26274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:19:06.108594 26270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72392e8ea62f4f31acbddad82a9f660d. Latest consensus state: current_term: 2 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:19:06.108739 26274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:06.108754 26270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:06.109138 26285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:06.109947 26285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:06.110047 26285 catalog_manager.cc:1259] Loaded cluster ID: 9c00dfc2b7c14c198c8d0ea729fd1195
I20260810 15:19:06.110070 26285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:06.111249 26285 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:06.111469 26285 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Loaded TSK: 0
I20260810 15:19:06.126910 26285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Generated new TSK 1
I20260810 15:19:06.127017 26285 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:19:06.181967 26264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:06.182168 26264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:06.182199 26264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:06.183337 26264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:06.183377 26264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:06.183413 26264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:06.184907 26264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:32967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=40001
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:06.185144 26264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:06.185349 26264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:06.190256 26310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.190243 26312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.190243 26305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:06.190649 26264 server_base.cc:1034] running on GCE node
I20260810 15:19:06.190840 26264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:06.191054 26264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:06.192222 26264 hybrid_clock.cc:648] HybridClock initialized: now 1786375146192198 us; error 40 us; skew 500 ppm
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:06.194909 26264 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:19:06.195492 26264 webserver.cc:460] Webserver started at http://127.29.230.1:40001/ using document root <none> and password file <none>
I20260810 15:19:06.195765 26264 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:06.195852 26264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:06.197595 26264 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:19:06.198366 26328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.198565 26264 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:19:06.198668 26264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "0452ba26ab5a4c7792404f8a3224c71b"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "a4b09acbbb2cc9b79b0d9d414ce4a146"
server_key_iv: "c2924cfb2f69850b7eaaa7ee49923c82"
server_key_version: "encryptionkey@0"
I20260810 15:19:06.199100 26264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:06.212543 26264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:06.215375 26264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:06.215528 26264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:06.215790 26264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:06.216331 26341 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:19:06.217978 26264 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:19:06.218072 26264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:19:06.218149 26264 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:19:06.219048 26264 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:19:06.219108 26264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:19:06.219158 26341 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap starting.
I20260810 15:19:06.226276 26264 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:32967
I20260810 15:19:06.226737 26264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:06.227599 26522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:32967 every 8 connection(s)
I20260810 15:19:06.229885 26341 log.cc:824] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:06.232736 26341 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:19:06.233047 26341 tablet_bootstrap.cc:492] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap complete.
I20260810 15:19:06.234102 26341 ts_tablet_manager.cc:1397] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:06.235798 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 26264
I20260810 15:19:06.236070 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:42281
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=43843
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:19:06.236202 26341 raft_consensus.cc:348] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:19:06.236709 26341 raft_consensus.cc:729] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452ba26ab5a4c7792404f8a3224c71b, State: Initialized, Role: FOLLOWER
I20260810 15:19:06.236965 26341 consensus_queue.cc:260] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } }
I20260810 15:19:06.237404 26341 ts_tablet_manager.cc:1428] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260810 15:19:06.240031 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.227638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:52973)
0810 15:19:06.227998 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.228048 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.228726 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.229348 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.229352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.229709 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:06.229970 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.229976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.231731 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.231733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.232984 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.232988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.233854 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.233856 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.233990 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.234548 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:06.234557 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:06.236248 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:06.238348 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.238356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.238358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.238643 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.238916 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.238917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.238917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.239045 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.239503 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:06.239509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:06.239636 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.239725 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.239865 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":94,"threads_started":1}
I20260810 15:19:06.240618 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.227718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:52973 (local address 127.29.230.62:34425)
0810 15:19:06.227892 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.227901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.228185 (+ 284us) server_negotiation.cc:407] Connection header received
0810 15:19:06.228973 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.228981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.229039 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.229197 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.230126 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.231603 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.233092 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.233655 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.233745 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.236359 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.236363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.236364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.236376 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.238258 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.238738 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.238741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.238741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.238849 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.239120 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.239121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.239121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.239413 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.239477 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.239883 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.240351 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260810 15:19:06.241323 26523 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:19:06.241415 26523 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:06.241585 26523 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:19:06.242386 26196 ts_manager.cc:194] Registered new tserver with Master: 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967)
I20260810 15:19:06.243563 26196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:52973
I20260810 15:19:06.246124 26523 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.1@KRBTEST.COM
I20260810 15:19:06.255398 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.243073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32967 (local address 127.0.0.1:46382)
0810 15:19:06.244140 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.244173 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.244231 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.245845 (+ 1614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.245851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.246271 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:06.246333 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.246336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.247517 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.247518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.248307 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.248311 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.249409 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.249411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.249496 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.250071 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:06.250077 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:06.251783 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:06.253858 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.253865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.253867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.254344 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.254617 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.254618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.254618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.254680 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.254969 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:06.254976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:06.255025 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.255110 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.255238 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":924,"threads_started":1}
I20260810 15:19:06.255808 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.244157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.29.230.1:32967)
0810 15:19:06.245677 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.245681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.245693 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:19:06.245726 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.245728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.245743 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.245820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.246422 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.247431 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.248882 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.249310 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.249700 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.251884 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.251893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.251894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.251901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.253713 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.254479 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.254482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.254482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.254546 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.254738 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.254738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.254739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.254884 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.254946 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.255628 (+ 682us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.255713 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1430,"threads_started":1}
I20260810 15:19:06.256706 26410 tablet_service.cc:1514] Processing DeleteTablet for tablet 87681121c92c4b40ba2ec0a0c3170db2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:18:45 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:46382
I20260810 15:19:06.258498 26534 tablet_replica.cc:330] stopping tablet replica
I20260810 15:19:06.258602 26534 raft_consensus.cc:2229] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:06.258678 26534 raft_consensus.cc:2258] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:06.259006 26534 ts_tablet_manager.cc:1905] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:06.261305 26534 ts_tablet_manager.cc:1918] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:06.261386 26534 log.cc:1196] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal/wals/87681121c92c4b40ba2ec0a0c3170db2
I20260810 15:19:06.261622 26534 ts_tablet_manager.cc:1939] T 87681121c92c4b40ba2ec0a0c3170db2 P 0452ba26ab5a4c7792404f8a3224c71b: Deleting consensus metadata
I20260810 15:19:06.262336 26168 catalog_manager.cc:4841] TS 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967): tablet 87681121c92c4b40ba2ec0a0c3170db2 (table test-table [id=9d00b1f75cbd418484c5e93b29f7781d]) successfully deleted
W20260810 15:19:06.336283 26530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:06.336504 26530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:06.336540 26530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:06.337646 26530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:06.337683 26530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:06.337718 26530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:06.339141 26530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:42281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=43843
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:06.339388 26530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:06.339603 26530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:06.344667 26549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.344645 26542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.344637 26543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:06.344920 26530 server_base.cc:1034] running on GCE node
I20260810 15:19:06.345124 26530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:06.345345 26530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:06.346483 26530 hybrid_clock.cc:648] HybridClock initialized: now 1786375146346467 us; error 28 us; skew 500 ppm
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:06.348933 26530 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:19:06.349371 26530 webserver.cc:460] Webserver started at http://127.29.230.2:43843/ using document root <none> and password file <none>
I20260810 15:19:06.349565 26530 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:06.349632 26530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:06.350898 26530 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:19:06.351471 26557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.351616 26530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.351699 26530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "a172a71ee42e411582a97fffcbfd3d05"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "d627f92330fd03f0db6706afe032ed15"
server_key_iv: "248158d7cf6422e0740515b4d0ded471"
server_key_version: "encryptionkey@0"
I20260810 15:19:06.352020 26530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:06.374810 26530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:06.376648 26530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:06.376782 26530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:06.376992 26530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:06.377291 26530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:06.377324 26530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.377353 26530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:06.377370 26530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.382673 26530 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42281
I20260810 15:19:06.382735 26739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42281 every 8 connection(s)
I20260810 15:19:06.383044 26530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:06.390895 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 26530
I20260810 15:19:06.391149 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:40511
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=40997
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:19:06.391343 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.383743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:53065 (local address 127.29.230.62:34425)
0810 15:19:06.383809 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.383811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.383996 (+ 185us) server_negotiation.cc:407] Connection header received
0810 15:19:06.384454 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.384455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.384468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.384513 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.385404 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.385905 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.386945 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.387137 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.387183 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.389092 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.389094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.389094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.389100 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.390151 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.390476 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.390477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.390477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.390521 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.390690 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.390691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.390691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.390781 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.390837 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.391140 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.391228 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:19:06.391485 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.383721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:53065)
0810 15:19:06.383882 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.383917 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.384376 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.384574 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.384578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.385133 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:06.385317 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.385322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.385972 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.385973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.386878 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.386882 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.387197 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.387198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.387329 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.387754 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:06.387762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:06.389009 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:06.390217 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.390224 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.390225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.390422 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.390562 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.390563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.390563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.390650 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.390849 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:06.390856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:06.390998 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.391108 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.391258 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260810 15:19:06.391921 26740 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:19:06.392028 26740 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:06.392200 26740 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:19:06.392565 26196 ts_manager.cc:194] Registered new tserver with Master: a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:19:06.393040 26196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:53065
W20260810 15:19:06.471792 26744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:06.471975 26744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:06.472004 26744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:06.473125 26744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:06.473163 26744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:06.473188 26744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:06.474594 26744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=40997
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:06.474766 26744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:06.474944 26744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:06.479676 26758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.479693 26757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:06.479772 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:06.479674 26744 server_base.cc:1034] running on GCE node
I20260810 15:19:06.480008 26744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:06.480202 26744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:06.481336 26744 hybrid_clock.cc:648] HybridClock initialized: now 1786375146481315 us; error 32 us; skew 500 ppm
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:06.483688 26744 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:19:06.484081 26744 webserver.cc:460] Webserver started at http://127.29.230.3:40997/ using document root <none> and password file <none>
I20260810 15:19:06.484289 26744 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:06.484349 26744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:06.485669 26744 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:06.486207 26778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.486377 26744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.486460 26744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "3d6af5c9b53e0ea8cb477845818110d2"
server_key_iv: "1c330f7488c2e3542fc9c86f8b9dd31a"
server_key_version: "encryptionkey@0"
I20260810 15:19:06.486796 26744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:06.509466 26744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:06.511305 26744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:06.511442 26744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:06.511675 26744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:06.511974 26744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:06.512009 26744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.512038 26744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:06.512056 26744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:06.517618 26744 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:40511
I20260810 15:19:06.517671 26984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:40511 every 8 connection(s)
I20260810 15:19:06.517997 26744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:19:06 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375146, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:06.526437 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.518695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:33031 (local address 127.29.230.62:34425)
0810 15:19:06.518745 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.518747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.518995 (+ 248us) server_negotiation.cc:407] Connection header received
0810 15:19:06.519461 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.519463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.519474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.519533 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.520404 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.520891 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.521854 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.522093 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.522156 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.524080 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.524081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.524082 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.524089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.525333 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.525667 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.525667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.525668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.525718 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.525886 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.525886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.525887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.525984 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.526026 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.526286 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.526365 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:19:06.526521 26987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.518672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:33031)
0810 15:19:06.518823 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.518864 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.519369 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.519587 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.519591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.520130 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:06.520325 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.520329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.520958 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.520959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.521756 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.521758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.522160 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.522161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.522253 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.522684 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:06.522691 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:06.523995 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:06.525400 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.525404 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.525406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.525608 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.525763 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:06.525763 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:06.525763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:06.525847 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:06.526031 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:06.526035 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:06.526147 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.526228 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.526342 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260810 15:19:06.526973 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 26744
I20260810 15:19:06.527064 26985 heartbeater.cc:347] Connected to a master server at 127.29.230.62:34425
I20260810 15:19:06.527172 26985 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:06.527395 26985 heartbeater.cc:511] Master 127.29.230.62:34425 requested a full tablet report, sending...
I20260810 15:19:06.527781 26196 ts_manager.cc:194] Registered new tserver with Master: 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511)
I20260810 15:19:06.528146 26196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:33031
I20260810 15:19:06.532963 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.527347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.29.230.62:34425)
0810 15:19:06.527409 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.527410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.527571 (+ 161us) server_negotiation.cc:407] Connection header received
0810 15:19:06.527627 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.527628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.527638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.527687 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.528275 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.528926 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.529644 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.529838 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.529893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.530916 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.530917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.530918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.530924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.531973 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.532299 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.532300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.532300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.532354 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.532513 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.532514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.532515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.532620 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.532777 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.532814 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.532880 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:19:06.533568 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:06.535881 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.533860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.29.230.62:34425)
0810 15:19:06.533896 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.533897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.534008 (+ 111us) server_negotiation.cc:407] Connection header received
0810 15:19:06.534060 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.534061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.534089 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.534141 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:06.534219 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.534652 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.535353 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.535534 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.535576 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.535700 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:06.535737 (+ 37us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 15:19:06.535840 (+ 103us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44100: token expired
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:19:06.535933 26527 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44100: token expired
W20260810 15:19:06.536883 26065 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.230.62:34425)
I20260810 15:19:06.542073 26527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.537144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.29.230.62:34425)
0810 15:19:06.537189 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.537190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.537198 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:19:06.537217 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.537217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.537224 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.537268 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:06.537706 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.538132 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.538809 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.538981 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.539038 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.540003 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:06.540004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:06.540004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:06.540010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:06.541210 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.541529 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.541530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.541530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.541573 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:06.541700 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:06.541700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:06.541700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:06.541783 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:06.541845 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.541958 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.542020 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:19:06.543226 26196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:06.544909 26196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:06.546895 26410 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb366c8c59f4647b0f6233817c7176e (DEFAULT_TABLE table=test-table [id=07b58f21f6b049248f8eaa2e694f539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:06.547107 26410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb366c8c59f4647b0f6233817c7176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:06.548714 26341 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap starting.
I20260810 15:19:06.549048 26341 tablet_bootstrap.cc:654] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:06.549556 26341 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: No bootstrap required, opened a new log
I20260810 15:19:06.549598 26341 ts_tablet_manager.cc:1397] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:06.549929 26341 raft_consensus.cc:348] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.550011 26341 raft_consensus.cc:374] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:06.550041 26341 raft_consensus.cc:729] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452ba26ab5a4c7792404f8a3224c71b, State: Initialized, Role: FOLLOWER
I20260810 15:19:06.550081 26341 consensus_queue.cc:260] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.550171 26341 ts_tablet_manager.cc:1428] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 15:19:06.550411 26992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.546650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42281 (local address 127.0.0.1:41482)
0810 15:19:06.546974 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.546985 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.547042 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.547323 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.547323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.547326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:06.547370 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.547371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.548248 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.548249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.549166 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.549169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.550168 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.550169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.550236 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.550260 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.550303 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.550342 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":71,"threads_started":1}
I20260810 15:19:06.550639 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.546855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.29.230.2:42281)
0810 15:19:06.547169 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.547173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.547183 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:06.547220 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.547222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.547237 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.547327 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:06.547417 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.548182 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.549259 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.550082 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.550412 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.550491 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.550519 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.550562 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":66,"threads_started":1}
I20260810 15:19:06.550704 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.546599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40511 (local address 127.0.0.1:51748)
0810 15:19:06.546641 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.546651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.546682 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.547190 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.547191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.547193 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:06.547234 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.547235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.548746 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.548747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.549646 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.549649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.550511 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.550512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.550577 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.550580 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.550609 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.550648 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:19:06.550992 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.546812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.29.230.3:40511)
0810 15:19:06.547004 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.547009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.547019 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:06.547055 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.547057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.547071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.547353 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:06.547436 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.548630 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.549739 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.550410 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.550781 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.550846 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.550870 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.550922 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260810 15:19:06.551378 26633 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb366c8c59f4647b0f6233817c7176e (DEFAULT_TABLE table=test-table [id=07b58f21f6b049248f8eaa2e694f539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:06.551613 26864 tablet_service.cc:1467] Processing CreateTablet for tablet 7cb366c8c59f4647b0f6233817c7176e (DEFAULT_TABLE table=test-table [id=07b58f21f6b049248f8eaa2e694f539c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:06.551739 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb366c8c59f4647b0f6233817c7176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:06.551910 26864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cb366c8c59f4647b0f6233817c7176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:06.555356 27000 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Bootstrap starting.
I20260810 15:19:06.555933 27000 tablet_bootstrap.cc:654] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:06.556420 27000 log.cc:824] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:06.556710 26999 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Bootstrap starting.
I20260810 15:19:06.557169 27000 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: No bootstrap required, opened a new log
I20260810 15:19:06.557243 27000 ts_tablet_manager.cc:1397] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:06.557345 26999 tablet_bootstrap.cc:654] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:06.557888 26999 log.cc:824] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:06.558650 26999 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: No bootstrap required, opened a new log
I20260810 15:19:06.558722 26999 ts_tablet_manager.cc:1397] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:06.559249 27000 raft_consensus.cc:348] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.559474 27000 raft_consensus.cc:374] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:06.559531 27000 raft_consensus.cc:729] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a172a71ee42e411582a97fffcbfd3d05, State: Initialized, Role: FOLLOWER
I20260810 15:19:06.559653 27000 consensus_queue.cc:260] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.559927 27000 ts_tablet_manager.cc:1428] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 15:19:06.559932 26740 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
I20260810 15:19:06.560490 26999 raft_consensus.cc:348] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.560639 26999 raft_consensus.cc:374] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:06.560685 26999 raft_consensus.cc:729] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe738f597584e02924ec6ee7fbd7b6d, State: Initialized, Role: FOLLOWER
I20260810 15:19:06.560797 26999 consensus_queue.cc:260] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.561072 26999 ts_tablet_manager.cc:1428] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:19:06.561100 26985 heartbeater.cc:503] Master 127.29.230.62:34425 was elected leader, sending a full tablet report...
W20260810 15:19:06.633675 26742 tablet.cc:2369] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:06.651651 26529 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:06.651791 26529 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.652112 26529 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:19:06.655542 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.652370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:40945 (local address 127.29.230.2:42281)
0810 15:19:06.652422 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.652424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.652654 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:19:06.652686 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.652687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.652698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.652749 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:06.653026 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.653550 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.654607 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.655156 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.655215 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.655222 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.655428 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.655468 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:19:06.655540 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.652363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:43761 (local address 127.29.230.3:40511)
0810 15:19:06.652437 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.652439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.652449 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:06.652555 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.652556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.652566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.652621 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:06.652791 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.653342 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.654530 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.655130 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.655208 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.655217 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.655426 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.655468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 15:19:06.655552 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.652302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40511 (local address 127.29.230.1:43761)
0810 15:19:06.652364 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.652377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.652445 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.652654 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.652656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.652658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:06.652721 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.652723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.653431 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.653432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.654443 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.654446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.655217 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.655218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.655322 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.655358 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.655418 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.655466 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":54}
I20260810 15:19:06.655618 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.652302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42281 (local address 127.29.230.1:40945)
0810 15:19:06.652527 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.652537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:06.652604 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:06.652772 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:06.652773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:06.652775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:06.652873 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.652875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.653646 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.653647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.654529 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:06.654532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.655241 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.655243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:06.655322 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.655358 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 15:19:06.655418 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 15:19:06.655481 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":51,"thread_start_us":111,"threads_started":1}
I20260810 15:19:06.655942 26671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb366c8c59f4647b0f6233817c7176e" candidate_uuid: "0452ba26ab5a4c7792404f8a3224c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a172a71ee42e411582a97fffcbfd3d05" is_pre_election: true
I20260810 15:19:06.655941 26901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb366c8c59f4647b0f6233817c7176e" candidate_uuid: "0452ba26ab5a4c7792404f8a3224c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" is_pre_election: true
I20260810 15:19:06.656102 26901 raft_consensus.cc:2454] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0452ba26ab5a4c7792404f8a3224c71b in term 0.
I20260810 15:19:06.656116 26671 raft_consensus.cc:2454] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0452ba26ab5a4c7792404f8a3224c71b in term 0.
I20260810 15:19:06.656327 26333 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b, a172a71ee42e411582a97fffcbfd3d05; no voters:
I20260810 15:19:06.656467 26529 raft_consensus.cc:2781] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:06.656522 26529 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:06.656538 26529 raft_consensus.cc:3037] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:06.657228 26529 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.657349 26529 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 1 election: Requested vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
I20260810 15:19:06.657523 26671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb366c8c59f4647b0f6233817c7176e" candidate_uuid: "0452ba26ab5a4c7792404f8a3224c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a172a71ee42e411582a97fffcbfd3d05"
I20260810 15:19:06.657524 26901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cb366c8c59f4647b0f6233817c7176e" candidate_uuid: "0452ba26ab5a4c7792404f8a3224c71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
I20260810 15:19:06.657599 26671 raft_consensus.cc:3037] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:06.657612 26901 raft_consensus.cc:3037] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:06.658267 26671 raft_consensus.cc:2454] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0452ba26ab5a4c7792404f8a3224c71b in term 1.
I20260810 15:19:06.658385 26901 raft_consensus.cc:2454] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0452ba26ab5a4c7792404f8a3224c71b in term 1.
I20260810 15:19:06.658439 26333 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b, a172a71ee42e411582a97fffcbfd3d05; no voters:
I20260810 15:19:06.658550 26529 raft_consensus.cc:2781] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:06.658710 26529 raft_consensus.cc:686] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 LEADER]: Becoming Leader. State: Replica: 0452ba26ab5a4c7792404f8a3224c71b, State: Running, Role: LEADER
I20260810 15:19:06.658777 26529 consensus_queue.cc:237] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:06.659616 26196 catalog_manager.cc:5495] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1). New cstate: current_term: 1 leader_uuid: "0452ba26ab5a4c7792404f8a3224c71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } health_report { overall_health: HEALTHY } } }
I20260810 15:19:06.665104 26532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.663025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.29.230.1:32967)
0810 15:19:06.663070 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:06.663071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:06.663079 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:19:06.663126 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:06.663127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:06.663142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:06.663198 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:06.663323 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.663778 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.664493 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:06.664697 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:06.664754 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:06.664845 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:06.664946 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:06.664996 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:06.665036 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 15:19:06.665056 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:19:06.666363 26671 raft_consensus.cc:1261] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Refusing update from remote peer 0452ba26ab5a4c7792404f8a3224c71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:06.666371 26901 raft_consensus.cc:1261] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Refusing update from remote peer 0452ba26ab5a4c7792404f8a3224c71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:06.666662 27015 consensus_queue.cc:1035] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:06.666797 26529 consensus_queue.cc:1035] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:06.669552 27023 mvcc.cc:205] Tried to move back new op lower bound from 7316992600743436288 to 7316992600715042816. Current Snapshot: MvccSnapshot[applied={T|T < 7316992600743436288}]
I20260810 15:19:06.669677 27022 mvcc.cc:205] Tried to move back new op lower bound from 7316992600743436288 to 7316992600715042816. Current Snapshot: MvccSnapshot[applied={T|T < 7316992600743436288 or (T in {7316992600743436288})}]
I20260810 15:19:06.671640 26196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:06.671721 26196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:06.672509 26196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Sending DeleteTablet for 3 replicas of tablet 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:06.672859 26410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb366c8c59f4647b0f6233817c7176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:06 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:46382
I20260810 15:19:06.672859 26633 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb366c8c59f4647b0f6233817c7176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:06 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:41482
I20260810 15:19:06.672863 26864 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cb366c8c59f4647b0f6233817c7176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:06 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:51748
I20260810 15:19:06.673045 26534 tablet_replica.cc:330] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: stopping tablet replica
I20260810 15:19:06.673137 26534 raft_consensus.cc:2229] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:06.673164 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 26264
I20260810 15:19:06.673218 26534 raft_consensus.cc:2258] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:06.695609 27038 tablet_replica.cc:330] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: stopping tablet replica
I20260810 15:19:06.695787 27038 raft_consensus.cc:2229] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:06.695923 27038 raft_consensus.cc:2258] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:06.696115 27039 tablet_replica.cc:330] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: stopping tablet replica
I20260810 15:19:06.696214 27039 raft_consensus.cc:2229] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:06.696290 27039 raft_consensus.cc:2258] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:06.702662 27039 ts_tablet_manager.cc:1905] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:06.702792 27038 ts_tablet_manager.cc:1905] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:06.704305 27039 ts_tablet_manager.cc:1918] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:06.704420 27038 ts_tablet_manager.cc:1918] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:06.704481 27039 log.cc:1196] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:06.704499 27038 log.cc:1196] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:06.704748 27039 ts_tablet_manager.cc:1939] T 7cb366c8c59f4647b0f6233817c7176e P a172a71ee42e411582a97fffcbfd3d05: Deleting consensus metadata
I20260810 15:19:06.704828 27038 ts_tablet_manager.cc:1939] T 7cb366c8c59f4647b0f6233817c7176e P 1fe738f597584e02924ec6ee7fbd7b6d: Deleting consensus metadata
I20260810 15:19:06.705253 26166 catalog_manager.cc:4841] TS a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): tablet 7cb366c8c59f4647b0f6233817c7176e (table test-table [id=07b58f21f6b049248f8eaa2e694f539c]) successfully deleted
I20260810 15:19:06.705273 26167 catalog_manager.cc:4841] TS 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): tablet 7cb366c8c59f4647b0f6233817c7176e (table test-table [id=07b58f21f6b049248f8eaa2e694f539c]) successfully deleted
W20260810 15:19:06.706859 26168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:32967 (error 108)
I20260810 15:19:06.707193 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 26530
I20260810 15:19:06.710592 26531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:06.707254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:32967 (local address 127.0.0.1:46400)
0810 15:19:06.710511 (+ 3257us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:06.710560 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:32967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3236}
W20260810 15:19:06.719518 26168 catalog_manager.cc:4584] TS 0452ba26ab5a4c7792404f8a3224c71b (127.29.230.1:32967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7cb366c8c59f4647b0f6233817c7176e: Network error: Client connection negotiation failed: client connection to 127.29.230.1:32967: connect: Connection refused (error 111)
I20260810 15:19:06.721596 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 26744
I20260810 15:19:06.746939 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 26136
I20260810 15:19:26.765367 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--webserver_interface=127.29.230.62
--webserver_port=33559
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:26.851099 27048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:26.851339 27048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:26.851367 27048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:26.852504 27048 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 15:19:26.852536 27048 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 15:19:26.852555 27048 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 15:19:26.852566 27048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:26.852576 27048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:26.852588 27048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:26.852599 27048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:26.852615 27048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:26.854079 27048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=33559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:26.854280 27048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:26.854455 27048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:26.859356 27059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:26.859337 27061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:26.859452 27048 server_base.cc:1034] running on GCE node
W20260810 15:19:26.859337 27063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:26.859817 27048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:26.860113 27048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:26.861266 27048 hybrid_clock.cc:648] HybridClock initialized: now 1786375166861246 us; error 33 us; skew 500 ppm
Aug 10 15:19:26 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375166, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:26.863788 27048 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:19:26.864251 27048 webserver.cc:460] Webserver started at http://127.29.230.62:33559/ using document root <none> and password file <none>
I20260810 15:19:26.864480 27048 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:26.864530 27048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:26.865847 27048 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:26.866590 27073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:26.866765 27048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:26.866854 27048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "72392e8ea62f4f31acbddad82a9f660d"
format_stamp: "Formatted at 2026-08-10 15:18:44 on dist-test-slave-gjrz"
server_key: "04e838a8718ac5e4a898ef4708a8eb35"
server_key_iv: "b7291b1a1ff1af471b363062a3c05f0a"
server_key_version: "encryptionkey@0"
I20260810 15:19:26.867200 27048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:26.874970 27048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:26.876670 27048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:26.876794 27048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:26.880745 27048 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:34425
I20260810 15:19:26.880781 27165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:34425 every 8 connection(s)
I20260810 15:19:26.881135 27048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:26.881287 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 27048
I20260810 15:19:26.881582 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:32967
--local_ip_for_outbound_sockets=127.29.230.1
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=40001
--webserver_interface=127.29.230.1
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:19:26.882678 27166 sys_catalog.cc:263] Verifying existing consensus state
I20260810 15:19:26.883370 27166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap starting.
I20260810 15:19:26.891566 27166 log.cc:824] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:26.893764 27166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:19:26.894052 27166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Bootstrap complete.
I20260810 15:19:26.896267 27166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:26.896502 27166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Initialized, Role: FOLLOWER
I20260810 15:19:26.896662 27166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:26.896741 27166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:26.896800 27166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:26.896848 27166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 2 FOLLOWER]: Advancing to term 3
I20260810 15:19:26.897835 27166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:26.897940 27166 leader_election.cc:302] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72392e8ea62f4f31acbddad82a9f660d; no voters:
I20260810 15:19:26.898105 27166 leader_election.cc:288] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 15:19:26.898170 27172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 3 FOLLOWER]: Leader election won for term 3
I20260810 15:19:26.898368 27166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:26.898386 27172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [term 3 LEADER]: Becoming Leader. State: Replica: 72392e8ea62f4f31acbddad82a9f660d, State: Running, Role: LEADER
I20260810 15:19:26.898478 27172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } }
I20260810 15:19:26.898799 27172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72392e8ea62f4f31acbddad82a9f660d. Latest consensus state: current_term: 3 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:19:26.898897 27173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "72392e8ea62f4f31acbddad82a9f660d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72392e8ea62f4f31acbddad82a9f660d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 34425 } } }
I20260810 15:19:26.899063 27173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:26.898957 27172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:26.899324 27189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:26.900167 27189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:26.900266 27189 catalog_manager.cc:1259] Loaded cluster ID: 9c00dfc2b7c14c198c8d0ea729fd1195
I20260810 15:19:26.900301 27189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:26.901451 27189 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:26.901647 27189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Loaded TSK: 0
I20260810 15:19:26.901697 27189 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72392e8ea62f4f31acbddad82a9f660d: Loaded TSK: 1
F20260810 15:19:26.901939 27189 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe71e0099ad google::LogMessage::Fail() at ??:0
@ 0x7fe71e00c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe71e0094de google::LogMessage::Flush() at ??:0
@ 0x7fe71e00aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe720776104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe720773d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe725f631cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe725f6348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe725f63594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe725f29c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe725f33ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe71fed70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe71fecc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe720d306db start_thread at ??:0
@ 0x7fe71cb0761f clone at ??:0
W20260810 15:19:26.980799 27170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:26.980998 27170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:26.981029 27170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:26.982156 27170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:26.982198 27170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:26.982234 27170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:26.983747 27170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:32967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=40001
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:26.983987 27170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:26.984200 27170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:26.990002 27235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:26.990002 27236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:26.990231 27170 server_base.cc:1034] running on GCE node
W20260810 15:19:26.990064 27238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:26.990528 27170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:26.990829 27170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:26.992002 27170 hybrid_clock.cc:648] HybridClock initialized: now 1786375166991978 us; error 37 us; skew 500 ppm
Aug 10 15:19:26 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375166, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:26.994907 27170 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:19:26.995510 27170 webserver.cc:460] Webserver started at http://127.29.230.1:40001/ using document root <none> and password file <none>
I20260810 15:19:26.995787 27170 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:26.995860 27170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:26.997568 27170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:26.998308 27253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:26.998479 27170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:26.998556 27170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "0452ba26ab5a4c7792404f8a3224c71b"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "a4b09acbbb2cc9b79b0d9d414ce4a146"
server_key_iv: "c2924cfb2f69850b7eaaa7ee49923c82"
server_key_version: "encryptionkey@0"
I20260810 15:19:26.998878 27170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:27.014909 27170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:27.017005 27170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:27.017148 27170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:27.017364 27170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:27.017807 27267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 15:19:27.019855 27170 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 15:19:27.019905 27170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 15:19:27.019938 27170 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 15:19:27.020570 27170 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 15:19:27.020604 27170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 15:19:27.020696 27267 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap starting.
I20260810 15:19:27.027486 27170 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:32967
I20260810 15:19:27.027623 27457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:32967 every 8 connection(s)
I20260810 15:19:27.027926 27170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:27.031659 27267 log.cc:824] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:27.034463 27267 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 15:19:27.034770 27267 tablet_bootstrap.cc:492] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Bootstrap complete.
I20260810 15:19:27.035871 27267 ts_tablet_manager.cc:1397] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260810 15:19:27.037500 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 27170
I20260810 15:19:27.037334 27267 raft_consensus.cc:348] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:27.037680 27267 raft_consensus.cc:729] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0452ba26ab5a4c7792404f8a3224c71b, State: Initialized, Role: FOLLOWER
I20260810 15:19:27.037774 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:42281
--local_ip_for_outbound_sockets=127.29.230.2
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=43843
--webserver_interface=127.29.230.2
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:19:27.037909 27267 consensus_queue.cc:260] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:27.038230 27267 ts_tablet_manager.cc:1428] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260810 15:19:27.110255 27462 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:19:27.028654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:43931)
0810 15:19:27.028829 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.028864 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:27.029297 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:27.110040 (+ 80743us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
W20260810 15:19:27.111577 27458 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:34425 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:34425: Client connection negotiation failed: client connection to 127.29.230.62:34425: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 15:19:27.114622 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.114461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:38911)
0810 15:19:27.114519 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.114582 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:19:27.115072 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.114972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:46777)
0810 15:19:27.115012 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.115047 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 15:19:27.115211 27458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 15:19:27.125968 27470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:27.126184 27470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:27.126224 27470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:27.127415 27470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:27.127465 27470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:27.127503 27470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:27.128983 27470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:42281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=43843
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:27.129230 27470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:27.129446 27470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:27.134542 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:27.134546 27470 server_base.cc:1034] running on GCE node
W20260810 15:19:27.134523 27480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:27.134531 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:27.135003 27470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:27.135251 27470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:27.136430 27470 hybrid_clock.cc:648] HybridClock initialized: now 1786375167136406 us; error 40 us; skew 500 ppm
Aug 10 15:19:27 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375167, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:27.139070 27470 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:19:27.139570 27470 webserver.cc:460] Webserver started at http://127.29.230.2:43843/ using document root <none> and password file <none>
I20260810 15:19:27.139786 27470 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:27.139838 27470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:27.141239 27470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:27.141842 27492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.142009 27470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.142102 27470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "a172a71ee42e411582a97fffcbfd3d05"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "d627f92330fd03f0db6706afe032ed15"
server_key_iv: "248158d7cf6422e0740515b4d0ded471"
server_key_version: "encryptionkey@0"
I20260810 15:19:27.142524 27470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:27.179309 27470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:27.181418 27470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:27.181566 27470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:27.181802 27470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:27.182133 27470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:27.182169 27470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.182200 27470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:27.182219 27470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.188114 27470 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42281
I20260810 15:19:27.188174 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42281 every 8 connection(s)
I20260810 15:19:27.188509 27470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:27.189575 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.189166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:53231)
0810 15:19:27.189342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.189420 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
W20260810 15:19:27.190028 27699 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:34425 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:34425: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
I20260810 15:19:27.190342 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.190229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:39477)
0810 15:19:27.190290 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.190320 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 15:19:27.190681 27703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.190560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:57313)
0810 15:19:27.190628 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.190659 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 15:19:27.190814 27699 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:19:27.195784 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 27470
I20260810 15:19:27.196049 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:40511
--local_ip_for_outbound_sockets=127.29.230.3
--tserver_master_addrs=127.29.230.62:34425
--webserver_port=40997
--webserver_interface=127.29.230.3
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:27.278512 27704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:27.278709 27704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:27.278739 27704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:27.279954 27704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:27.279997 27704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:27.280033 27704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:27.281483 27704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=40997
--tserver_master_addrs=127.29.230.62:34425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:27.281718 27704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:27.281956 27704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:27.286878 27713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:27.286885 27715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:27.287098 27704 server_base.cc:1034] running on GCE node
W20260810 15:19:27.286947 27717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:27.287391 27704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:27.287667 27704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:27.288805 27704 hybrid_clock.cc:648] HybridClock initialized: now 1786375167288778 us; error 43 us; skew 500 ppm
Aug 10 15:19:27 dist-test-slave-gjrz krb5kdc[25166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375167, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:27.291183 27704 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:19:27.291661 27704 webserver.cc:460] Webserver started at http://127.29.230.3:40997/ using document root <none> and password file <none>
I20260810 15:19:27.291878 27704 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:27.291927 27704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:27.293239 27704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:27.293781 27731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.293939 27704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:27.294019 27704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "1fe738f597584e02924ec6ee7fbd7b6d"
format_stamp: "Formatted at 2026-08-10 15:18:45 on dist-test-slave-gjrz"
server_key: "3d6af5c9b53e0ea8cb477845818110d2"
server_key_iv: "1c330f7488c2e3542fc9c86f8b9dd31a"
server_key_version: "encryptionkey@0"
I20260810 15:19:27.294350 27704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:27.312943 27704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:27.314870 27704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:27.315011 27704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:27.315231 27704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:27.315544 27704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:27.315579 27704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.315608 27704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:27.315631 27704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:27.321159 27704 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:40511
I20260810 15:19:27.321229 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:40511 every 8 connection(s)
I20260810 15:19:27.321535 27704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:27.321715 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 27704
I20260810 15:19:27.322620 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.322228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:52231)
0810 15:19:27.322402 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.322474 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260810 15:19:27.322985 27953 heartbeater.cc:644] Failed to heartbeat to 127.29.230.62:34425 (0 consecutive failures): Network error: Failed to ping master at 127.29.230.62:34425: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
I20260810 15:19:27.323235 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.323155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:33089)
0810 15:19:27.323193 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.323217 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:19:27.323470 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:27.323410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:42221)
0810 15:19:27.323439 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:27.323457 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 15:19:27.323555 27953 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 15:19:28.115844 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.115623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:35391)
0810 15:19:28.115740 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:28.115780 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260810 15:19:28.191426 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.191155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:52447)
0810 15:19:28.191330 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:28.191371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":61,"threads_started":1}
I20260810 15:19:28.324107 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.323856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:44033)
0810 15:19:28.324016 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:28.324054 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260810 15:19:28.679790 27963 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:28.679913 27963 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:28.680227 27963 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
Aug 10 15:19:28 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375166, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.3@KRBTEST.COM
Aug 10 15:19:28 dist-test-slave-gjrz krb5kdc[25166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375166, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.2@KRBTEST.COM
I20260810 15:19:28.690555 27966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.680413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42281 (local address 127.29.230.1:41543)
0810 15:19:28.680519 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:28.680529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:28.680613 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:28.681430 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:28.681432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:28.681821 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:28.682035 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:28.682041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.683144 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.683145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:28.684030 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:28.684033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.684616 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:28.684737 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:28.685253 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:28.685262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:28.687207 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:28.688972 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:28.688973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:28.688974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:28.689275 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:28.689512 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:28.689513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:28.689636 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:28.690037 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:28.690039 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:28.690157 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 15:19:28.690248 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:19:28.690453 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":54,"thread_start_us":52,"threads_started":1}
I20260810 15:19:28.690567 27967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.680520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:41543 (local address 127.29.230.2:42281)
0810 15:19:28.680703 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:28.680714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:28.680753 (+ 39us) server_negotiation.cc:407] Connection header received
0810 15:19:28.681223 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:28.681226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:28.681275 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:28.681398 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:28.682154 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.683023 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684118 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.684525 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684760 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:28.687353 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:28.687356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:28.687357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:28.687367 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:28.688888 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689362 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:28.689365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:28.689365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:28.689451 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689706 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:28.689707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:28.689708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:28.689967 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:28.690040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:28.690308 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 15:19:28.690441 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":57,"threads_started":1}
I20260810 15:19:28.690564 27969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.680494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:58307 (local address 127.29.230.3:40511)
0810 15:19:28.680703 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:28.680714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:28.680748 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:19:28.681197 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:28.681199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:28.681243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:28.681360 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:28.682159 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.683051 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684122 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.684524 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684645 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:28.687087 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:28.687092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:28.687092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:28.687103 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:28.688845 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689356 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:28.689360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:28.689361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:28.689452 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689694 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:28.689695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:28.689695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:28.689949 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:28.690027 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:28.690311 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 15:19:28.690436 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260810 15:19:28.690650 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:28.680370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:40511 (local address 127.29.230.1:58307)
0810 15:19:28.680502 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:28.680513 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:28.680579 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:28.681429 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:28.681432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:28.681819 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:28.682035 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:28.682041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.683217 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.683218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:28.684030 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:28.684033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:28.684614 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:28.684615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:28.684739 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:28.685293 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:28.685298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:28.686848 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:28.688935 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:28.688943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:28.688946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:28.689275 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:28.689521 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:28.689522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:28.689522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:28.689636 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:28.690018 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:28.690026 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:28.690157 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 15:19:28.690248 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:19:28.690532 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":189,"thread_start_us":55,"threads_started":1}
W20260810 15:19:28.692005 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:28.692093 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:28.692165 27259 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:28.692276 27963 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:29.116415 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:29.116274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:51579)
0810 15:19:29.116336 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:29.116373 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 15:19:29.192044 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:29.191840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:50805)
0810 15:19:29.191966 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:29.192006 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":54,"threads_started":1}
I20260810 15:19:29.324909 27973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:29.324702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:34817)
0810 15:19:29.324835 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:29.324869 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260810 15:19:30.117199 27977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:30.116946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:60503)
0810 15:19:30.117088 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:30.117132 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260810 15:19:30.192715 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:30.192509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:59573)
0810 15:19:30.192630 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:30.192670 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260810 15:19:30.325608 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:30.325404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:51351)
0810 15:19:30.325525 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:30.325565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260810 15:19:30.333079 27983 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:30.333171 27983 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:30.333346 27983 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
W20260810 15:19:30.333724 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:30.333815 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:30.333943 27261 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:30.334017 27983 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:31.117980 27984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:31.117737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:56693)
0810 15:19:31.117882 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:31.117931 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260810 15:19:31.193524 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:31.193270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:59403)
0810 15:19:31.193432 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:31.193480 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260810 15:19:31.326346 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:31.326132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:35737)
0810 15:19:31.326257 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:31.326304 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260810 15:19:32.118785 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:32.118555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:53899)
0810 15:19:32.118690 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:32.118732 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260810 15:19:32.194199 27992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:32.193995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:60817)
0810 15:19:32.194117 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:32.194153 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260810 15:19:32.326972 27993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:32.326778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:45575)
0810 15:19:32.326892 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:32.326928 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260810 15:19:32.333097 27994 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:32.333182 27994 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:32.333318 27994 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
W20260810 15:19:32.333664 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:32.333743 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:32.333786 27259 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:32.333851 27994 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:33.119616 27996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:33.119330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:45209)
0810 15:19:33.119504 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:33.119556 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260810 15:19:33.194828 27997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:33.194619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:32897)
0810 15:19:33.194747 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:33.194785 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260810 15:19:33.327683 28000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:33.327431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:44429)
0810 15:19:33.327589 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:33.327636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260810 15:19:34.120296 28001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:34.120054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:58225)
0810 15:19:34.120207 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:34.120247 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260810 15:19:34.195525 28002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:34.195297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:40713)
0810 15:19:34.195440 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:34.195483 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260810 15:19:34.328450 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:34.328180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:36969)
0810 15:19:34.328351 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:34.328398 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260810 15:19:35.121021 28005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:35.120759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:56529)
0810 15:19:35.120931 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:35.120971 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260810 15:19:35.196216 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:35.195965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:53157)
0810 15:19:35.196137 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:35.196174 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
I20260810 15:19:35.329141 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:35.328916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:37829)
0810 15:19:35.329061 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:35.329098 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260810 15:19:35.725060 28008 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:35.725193 28008 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:35.725364 28008 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
W20260810 15:19:35.725735 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:35.725823 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:35.725867 27259 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:35.725929 28008 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:36.121726 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:36.121481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:51695)
0810 15:19:36.121637 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:36.121675 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260810 15:19:36.196851 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:36.196662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:56491)
0810 15:19:36.196773 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:36.196809 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260810 15:19:36.329793 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:36.329595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:59367)
0810 15:19:36.329711 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:36.329749 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260810 15:19:37.122442 28015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:37.122196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:33467)
0810 15:19:37.122348 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:37.122389 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260810 15:19:37.197517 28016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:37.197273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:43065)
0810 15:19:37.197421 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:37.197471 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260810 15:19:37.330394 28018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:37.330192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:57495)
0810 15:19:37.330308 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:37.330342 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260810 15:19:38.123179 28023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:38.122949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:56039)
0810 15:19:38.123086 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:38.123132 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260810 15:19:38.198159 28025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:38.197915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:58445)
0810 15:19:38.198053 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:38.198112 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260810 15:19:38.331054 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:38.330834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:54111)
0810 15:19:38.330969 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:38.331009 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260810 15:19:38.536772 28027 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:38.536873 28027 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:38.537019 28027 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
W20260810 15:19:38.537434 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:38.537534 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:38.537569 27259 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:38.537632 28027 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:39.123992 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:39.123697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:60221)
0810 15:19:39.123836 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:39.123876 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260810 15:19:39.198750 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:39.198553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:42115)
0810 15:19:39.198668 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:39.198704 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260810 15:19:39.331701 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:39.331502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:42861)
0810 15:19:39.331621 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:39.331657 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260810 15:19:40.124683 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:40.124458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:49033)
0810 15:19:40.124605 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:40.124635 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260810 15:19:40.199425 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:40.199162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:33989)
0810 15:19:40.199331 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:40.199373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":68,"threads_started":1}
I20260810 15:19:40.332442 28036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:40.332215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:48927)
0810 15:19:40.332348 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:40.332395 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260810 15:19:41.125422 28037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:41.125192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:43835)
0810 15:19:41.125331 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:41.125372 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260810 15:19:41.200089 28038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:41.199828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:51587)
0810 15:19:41.199976 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:41.200017 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260810 15:19:41.333137 28039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:41.332925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:49649)
0810 15:19:41.333051 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:41.333090 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260810 15:19:41.687119 28040 raft_consensus.cc:482] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:41.687229 28040 raft_consensus.cc:504] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fe738f597584e02924ec6ee7fbd7b6d" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 40511 } } peers { permanent_uuid: "a172a71ee42e411582a97fffcbfd3d05" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42281 } } peers { permanent_uuid: "0452ba26ab5a4c7792404f8a3224c71b" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 32967 } }
I20260810 15:19:41.687434 28040 leader_election.cc:288] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511), a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281)
W20260810 15:19:41.687863 27261 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1fe738f597584e02924ec6ee7fbd7b6d (127.29.230.3:40511): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
W20260810 15:19:41.687959 27259 leader_election.cc:341] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a172a71ee42e411582a97fffcbfd3d05 (127.29.230.2:42281): Not found: Tablet not found: 7cb366c8c59f4647b0f6233817c7176e
I20260810 15:19:41.688036 27259 leader_election.cc:302] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0452ba26ab5a4c7792404f8a3224c71b; no voters: 1fe738f597584e02924ec6ee7fbd7b6d, a172a71ee42e411582a97fffcbfd3d05
I20260810 15:19:41.688119 28040 raft_consensus.cc:2726] T 7cb366c8c59f4647b0f6233817c7176e P 0452ba26ab5a4c7792404f8a3224c71b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 15:19:42.126251 28042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:42.125957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.1:52055)
0810 15:19:42.126075 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:42.126113 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260810 15:19:42.200809 28044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:42.200618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.2:59929)
0810 15:19:42.200728 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:42.200764 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260810 15:19:42.331472 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 27170
I20260810 15:19:42.341681 28047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:42.339396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:34425 (local address 127.29.230.3:38773)
0810 15:19:42.341588 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:42.341626 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.62:34425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2173,"thread_start_us":2101,"threads_started":1}
I20260810 15:19:42.348863 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 27470
I20260810 15:19:42.357694 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 27704
2026-08-10T15:19:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58870 ms)
[----------] 30 tests from SecurityITest (235223 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:42 dist-test-slave-gjrz krb5kdc[28051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:42 dist-test-slave-gjrz krb5kdc[28051](info): set up 2 sockets
Aug 10 15:19:42 dist-test-slave-gjrz krb5kdc[28051](info): commencing operation
krb5kdc: starting...
W20260810 15:19:43.557343 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:43 dist-test-slave-gjrz krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:19:43.673635 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40377
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:43.755430 28077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:43.755640 28077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:43.755662 28077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:43.756803 28077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:43.756844 28077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:43.756856 28077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:43.756867 28077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:43.756887 28077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:43.758420 28077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:43.758675 28077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:43.758885 28077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:43.763895 28092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:43.763914 28093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:43.763960 28077 server_base.cc:1034] running on GCE node
W20260810 15:19:43.763912 28099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:43.764397 28077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:43.764714 28077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:43.765872 28077 hybrid_clock.cc:648] HybridClock initialized: now 1786375183765856 us; error 29 us; skew 500 ppm
Aug 10 15:19:43 dist-test-slave-gjrz krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375183, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:43.768508 28077 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:19:43.768937 28077 webserver.cc:460] Webserver started at http://127.29.230.62:44571/ using document root <none> and password file <none>
I20260810 15:19:43.769134 28077 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:43.769173 28077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:43.769259 28077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:43.770282 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "90f81fe675994191a3401aa15d4d8864"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "55cb7a0783428464d355763aa1bddb03"
server_key_iv: "9d6cfacd4a4d0eef46bfae12a0452da7"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.770574 28077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "90f81fe675994191a3401aa15d4d8864"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "55cb7a0783428464d355763aa1bddb03"
server_key_iv: "9d6cfacd4a4d0eef46bfae12a0452da7"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.771804 28077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:19:43.772588 28115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:43.772771 28077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:43.772867 28077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "90f81fe675994191a3401aa15d4d8864"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "55cb7a0783428464d355763aa1bddb03"
server_key_iv: "9d6cfacd4a4d0eef46bfae12a0452da7"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.772979 28077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:43.812793 28077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:43.814747 28077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:43.814893 28077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:43.818784 28077 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:40377
I20260810 15:19:43.818848 28203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:40377 every 8 connection(s)
I20260810 15:19:43.819188 28077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:43.820124 28207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:43.822494 28207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: Bootstrap starting.
I20260810 15:19:43.823103 28207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:43.823395 28207 log.cc:824] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:43.824090 28207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: No bootstrap required, opened a new log
I20260810 15:19:43.825661 28207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } }
I20260810 15:19:43.825800 28207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:43.825821 28207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f81fe675994191a3401aa15d4d8864, State: Initialized, Role: FOLLOWER
I20260810 15:19:43.825961 28207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } }
I20260810 15:19:43.826028 28207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:43.826050 28207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:43.826099 28207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:43.826604 28207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } }
I20260810 15:19:43.826695 28207 leader_election.cc:302] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90f81fe675994191a3401aa15d4d8864; no voters:
I20260810 15:19:43.826855 28207 leader_election.cc:288] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:43.826893 28210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:43.827032 28210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [term 1 LEADER]: Becoming Leader. State: Replica: 90f81fe675994191a3401aa15d4d8864, State: Running, Role: LEADER
I20260810 15:19:43.827104 28207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:43.827148 28210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } }
I20260810 15:19:43.827538 28214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90f81fe675994191a3401aa15d4d8864. Latest consensus state: current_term: 1 leader_uuid: "90f81fe675994191a3401aa15d4d8864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } } }
I20260810 15:19:43.827625 28214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:43.827979 28222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:43.828125 28213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90f81fe675994191a3401aa15d4d8864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f81fe675994191a3401aa15d4d8864" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40377 } } }
I20260810 15:19:43.828272 28213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:43.828614 28222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:43.829139 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 28077
I20260810 15:19:43.829238 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:43.829434 30616 external_mini_cluster.cc:1428] Setting key 7fe1502da968ae4ef97f5c108b97f129
I20260810 15:19:43.830096 28222 catalog_manager.cc:1347] Generated new cluster ID: aa1ea82207e740e8974e08223124f9e8
I20260810 15:19:43.830147 28222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:19:43 dist-test-slave-gjrz krb5kdc[28051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:43.842046 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:43.829871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.29.230.62:40377)
0810 15:19:43.830076 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:43.830085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:43.831434 (+ 1349us) server_negotiation.cc:407] Connection header received
0810 15:19:43.831576 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:43.831584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:43.831642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:43.831799 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:43.832492 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:43.833928 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:43.834968 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:43.835657 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:43.835775 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:43.838527 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:43.838531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:43.838532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:43.838545 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:43.840396 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:43.840788 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:43.840791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:43.840791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:43.840886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:43.841073 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:43.841073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:43.841074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:43.841344 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:43.841460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:43.841576 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 15:19:43.841794 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260810 15:19:43.848661 28222 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:43.849149 28222 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:43.859153 28222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: Generated new TSK 0
I20260810 15:19:43.859373 28222 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:19:43.888271 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:43.969589 28250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:43.969786 28250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:43.969818 28250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:43.970928 28250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:43.970970 28250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:43.971006 28250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:43.972492 28250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:43.972724 28250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:43.972929 28250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:43.977831 28257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:43.977823 28263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:43.977823 28259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:43.978118 28250 server_base.cc:1034] running on GCE node
I20260810 15:19:43.978307 28250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:43.978549 28250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:43.979691 28250 hybrid_clock.cc:648] HybridClock initialized: now 1786375183979675 us; error 33 us; skew 500 ppm
Aug 10 15:19:43 dist-test-slave-gjrz krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375183, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:43.982625 28250 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:19:43.983142 28250 webserver.cc:460] Webserver started at http://127.29.230.1:35711/ using document root <none> and password file <none>
I20260810 15:19:43.983402 28250 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:43.983469 28250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:43.983603 28250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:43.984663 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "1b3f8be114404b479c2ae870b18ad5e7"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "5847cc74ec351d49ca6a9ebb1624fb7f"
server_key_iv: "f9b6d52192abeb01dab0769ebfc145ee"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.984982 28250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "1b3f8be114404b479c2ae870b18ad5e7"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "5847cc74ec351d49ca6a9ebb1624fb7f"
server_key_iv: "f9b6d52192abeb01dab0769ebfc145ee"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.986265 28250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:43.986955 28277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:43.987125 28250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:43.987211 28250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "1b3f8be114404b479c2ae870b18ad5e7"
format_stamp: "Formatted at 2026-08-10 15:19:43 on dist-test-slave-gjrz"
server_key: "5847cc74ec351d49ca6a9ebb1624fb7f"
server_key_iv: "f9b6d52192abeb01dab0769ebfc145ee"
server_key_version: "encryptionkey@0"
I20260810 15:19:43.987305 28250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:43.999030 28250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.000701 28250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.000828 28250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.001024 28250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:44.001299 28250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:44.001332 28250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.001358 28250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:44.001394 28250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.006901 28250 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:42799
I20260810 15:19:44.006961 28495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:42799 every 8 connection(s)
I20260810 15:19:44.007233 28250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:19:44 dist-test-slave-gjrz krb5kdc[28051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375183, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:44.013578 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 28250
I20260810 15:19:44.013685 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:44.013882 30616 external_mini_cluster.cc:1428] Setting key 726de65ec61f3763e040b4913c0ed155
I20260810 15:19:44.016347 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.008005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:33005 (local address 127.29.230.62:40377)
0810 15:19:44.008107 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.008109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.008262 (+ 153us) server_negotiation.cc:407] Connection header received
0810 15:19:44.008750 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.008752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.008765 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.008816 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.009759 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.010295 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.011306 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.011522 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.011579 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.013583 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.013584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.013585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:44.013592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.015039 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.015465 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.015467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.015467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.015526 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.015740 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.015741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.015741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.015852 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.015907 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.016158 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.016257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 15:19:44.016484 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.007912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40377 (local address 127.29.230.1:33005)
0810 15:19:44.008130 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.008183 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.008653 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.008884 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.008888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.009481 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:44.009664 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.009670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.010375 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.010377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.011192 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.011194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.011597 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.011598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.011718 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.012110 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:44.012117 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:44.013488 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:44.015113 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.015120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.015122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.015384 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.015579 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.015580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.015580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.015684 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.015905 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:44.015912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:44.016042 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.016128 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.016273 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260810 15:19:44.017118 28499 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40377
I20260810 15:19:44.017197 28499 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:44.017354 28499 heartbeater.cc:511] Master 127.29.230.62:40377 requested a full tablet report, sending...
I20260810 15:19:44.017885 28144 ts_manager.cc:194] Registered new tserver with Master: 1b3f8be114404b479c2ae870b18ad5e7 (127.29.230.1:42799)
I20260810 15:19:44.018695 28144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:33005
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:19:44.039958 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:44.120765 28508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:44.120965 28508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:44.120993 28508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:44.122078 28508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:44.122112 28508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:44.122136 28508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:44.123538 28508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:44.123728 28508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:44.123906 28508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:44.128763 28525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.128787 28530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.128778 28523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.129052 28508 server_base.cc:1034] running on GCE node
I20260810 15:19:44.129246 28508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:44.129487 28508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:44.130647 28508 hybrid_clock.cc:648] HybridClock initialized: now 1786375184130632 us; error 28 us; skew 500 ppm
Aug 10 15:19:44 dist-test-slave-gjrz krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375184, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:44.133104 28508 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:19:44.133551 28508 webserver.cc:460] Webserver started at http://127.29.230.2:33257/ using document root <none> and password file <none>
I20260810 15:19:44.133764 28508 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:44.133814 28508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:44.134002 28508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:44.135167 28508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "79c5ac08892144fb9f811e2c69e40dd3"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "eeab3b4dcb7c958011f9bfb3637dec20"
server_key_iv: "101a4228177c350a22dc6e071408429a"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.135540 28508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "79c5ac08892144fb9f811e2c69e40dd3"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "eeab3b4dcb7c958011f9bfb3637dec20"
server_key_iv: "101a4228177c350a22dc6e071408429a"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.136988 28508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:44.137737 28549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.137940 28508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:44.138036 28508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "79c5ac08892144fb9f811e2c69e40dd3"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "eeab3b4dcb7c958011f9bfb3637dec20"
server_key_iv: "101a4228177c350a22dc6e071408429a"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.138105 28508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:44.148619 28508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.150424 28508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.150565 28508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.150779 28508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:44.151084 28508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:44.151124 28508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.151153 28508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:44.151176 28508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.156653 28508 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:38593
I20260810 15:19:44.156723 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:38593 every 8 connection(s)
I20260810 15:19:44.156989 28508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
Aug 10 15:19:44 dist-test-slave-gjrz krb5kdc[28051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375184, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:44.165251 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 28508
I20260810 15:19:44.165360 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:44.165580 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.157710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:48975 (local address 127.29.230.62:40377)
0810 15:19:44.157770 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.157773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.157942 (+ 169us) server_negotiation.cc:407] Connection header received
0810 15:19:44.158448 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.158449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.158462 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.158515 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.159464 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.159966 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.160948 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.161149 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.161205 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.163167 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.163168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.163169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:44.163176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.164300 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.164665 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.164666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.164667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.164721 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.164926 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.164927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.164928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.165047 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.165105 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.165387 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.165489 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:19:44.165587 30616 external_mini_cluster.cc:1428] Setting key c4811167e156bfaa3bd395994957c60a
I20260810 15:19:44.165669 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.157654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40377 (local address 127.29.230.2:48975)
0810 15:19:44.157821 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.157860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.158344 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.158587 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.158591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.159147 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:44.159367 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.159372 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.160056 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.160057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.160860 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.160862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.161217 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.161218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.161339 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.161775 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:44.161783 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:44.163070 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:44.164372 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.164378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.164381 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.164597 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.164774 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.164775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.164775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.164871 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.165102 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:44.165108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:44.165237 (+ 129us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.165329 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.165500 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260810 15:19:44.166153 28784 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40377
I20260810 15:19:44.166257 28784 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:44.166431 28784 heartbeater.cc:511] Master 127.29.230.62:40377 requested a full tablet report, sending...
I20260810 15:19:44.166790 28144 ts_manager.cc:194] Registered new tserver with Master: 79c5ac08892144fb9f811e2c69e40dd3 (127.29.230.2:38593)
I20260810 15:19:44.167151 28144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:48975
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:19:44.190287 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:44.270061 28797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:44.270255 28797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:44.270284 28797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:44.271411 28797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:44.271450 28797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:44.271490 28797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:44.272941 28797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:40377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:44.273177 28797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:44.273378 28797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:19:44.278525 28797 server_base.cc:1034] running on GCE node
W20260810 15:19:44.278496 28806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.278507 28812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.278496 28808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.278879 28797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:44.279117 28797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:44.280275 28797 hybrid_clock.cc:648] HybridClock initialized: now 1786375184280252 us; error 36 us; skew 500 ppm
Aug 10 15:19:44 dist-test-slave-gjrz krb5kdc[28051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375184, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:44.282745 28797 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:19:44.283191 28797 webserver.cc:460] Webserver started at http://127.29.230.3:39723/ using document root <none> and password file <none>
I20260810 15:19:44.283421 28797 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:44.283465 28797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:44.283557 28797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:44.284500 28797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "7fb44b5958504016823b527dcaf3cb26"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "9a0062fa6dd7a2ad47ab7fe9ea655c64"
server_key_iv: "3e2a1fe81d735b4d8671b02d89e85a4c"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.284788 28797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "7fb44b5958504016823b527dcaf3cb26"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "9a0062fa6dd7a2ad47ab7fe9ea655c64"
server_key_iv: "3e2a1fe81d735b4d8671b02d89e85a4c"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.286084 28797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:44.286759 28823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.286926 28797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:44.287014 28797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "7fb44b5958504016823b527dcaf3cb26"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "9a0062fa6dd7a2ad47ab7fe9ea655c64"
server_key_iv: "3e2a1fe81d735b4d8671b02d89e85a4c"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.287075 28797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:44.327370 28797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.329638 28797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.329751 28797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.329936 28797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:44.330201 28797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:44.330231 28797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.330259 28797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:44.330276 28797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.335842 28797 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:43489
I20260810 15:19:44.335898 29047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:43489 every 8 connection(s)
I20260810 15:19:44.336197 28797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
Aug 10 15:19:44 dist-test-slave-gjrz krb5kdc[28051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375184, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:44.345216 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.336851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:45771 (local address 127.29.230.62:40377)
0810 15:19:44.336929 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.336932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.337153 (+ 221us) server_negotiation.cc:407] Connection header received
0810 15:19:44.337643 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.337645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.337656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.337715 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.338690 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.339193 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.340236 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.340423 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.340478 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.342804 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.342806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.342806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:44.342813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.344113 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.344483 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.344483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.344483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.344530 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.344737 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.344738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.344738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.344835 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.344879 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.345088 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.345149 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 15:19:44.345381 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.336822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:40377 (local address 127.29.230.3:45771)
0810 15:19:44.337033 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.337070 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.337537 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.337778 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.337781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.338399 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:44.338597 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.338602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.339299 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.339300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.340160 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.340163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.340501 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.340503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.340617 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.341174 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:44.341182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:44.342700 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:44.344202 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.344208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.344210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.344422 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.344584 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:44.344585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:44.344585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:44.344690 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:44.344878 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:44.344881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:44.344990 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.345070 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.345188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":124,"threads_started":1}
I20260810 15:19:44.345793 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 28797
I20260810 15:19:44.345896 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:44.345897 29049 heartbeater.cc:347] Connected to a master server at 127.29.230.62:40377
I20260810 15:19:44.345997 29049 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:44.346091 30616 external_mini_cluster.cc:1428] Setting key b02a48d047fd88876d8155c3c04f764e
I20260810 15:19:44.346196 29049 heartbeater.cc:511] Master 127.29.230.62:40377 requested a full tablet report, sending...
I20260810 15:19:44.346577 28144 ts_manager.cc:194] Registered new tserver with Master: 7fb44b5958504016823b527dcaf3cb26 (127.29.230.3:43489)
I20260810 15:19:44.346908 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:44.347028 28144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:45771
I20260810 15:19:44.353349 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.348287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.29.230.62:40377)
0810 15:19:44.348346 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.348348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.348435 (+ 87us) server_negotiation.cc:407] Connection header received
0810 15:19:44.348473 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.348474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.348482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.348522 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.348979 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.349429 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.350141 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.350333 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.350396 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.351362 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.351364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.351364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:44.351370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.352566 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.352820 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.352821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.352821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.352863 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:44.352980 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:44.352980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:44.352981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:44.353099 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.353168 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.353236 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.353290 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:19:44.354657 28144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:44.356303 28144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:44.362607 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.29.230.1:42799)
0810 15:19:44.359149 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.359153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.359162 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:44.359199 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.359201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.359214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.359356 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:44.359461 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360368 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.361476 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.362210 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.362287 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.362351 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.362483 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.362524 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":83,"threads_started":1}
I20260810 15:19:44.362623 29063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:38593 (local address 127.0.0.1:36508)
0810 15:19:44.358573 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.358583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.358624 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.359002 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.359002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.359005 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:44.359053 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.359055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360146 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.360147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.361521 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.361524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.362365 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.362365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.362416 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.362420 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.362466 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.362533 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":42,"thread_start_us":70,"threads_started":1}
I20260810 15:19:44.362861 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.29.230.2:38593)
0810 15:19:44.358857 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.358860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.358869 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:44.358904 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.358906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.358920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.358996 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:44.359135 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360032 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.361602 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.362296 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.362358 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.362404 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.362741 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.362791 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":86,"threads_started":1}
I20260810 15:19:44.362622 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:42799 (local address 127.0.0.1:57460)
0810 15:19:44.358650 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.358658 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.358696 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.359358 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.359359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.359361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:44.359398 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.359399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360555 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.360556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.361395 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.361398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.362299 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.362300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.362371 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.362386 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.362463 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.362533 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":31,"thread_start_us":53,"threads_started":1}
I20260810 15:19:44.363409 29068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:43489 (local address 127.0.0.1:44782)
0810 15:19:44.358374 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.358414 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.358480 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.358777 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.358780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.358782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:44.358836 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.358837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360119 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.360120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.361520 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.361524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.363142 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.363143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.363208 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.363212 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.363296 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.363342 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":70,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
I20260810 15:19:44.363617 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.358444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.29.230.3:43489)
0810 15:19:44.358622 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.358627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.358637 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:44.358672 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.358673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.358685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.358755 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:44.358908 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.360018 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.362435 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.363064 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.363119 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.363156 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.363492 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.363538 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260810 15:19:44.363783 28656 tablet_service.cc:1467] Processing CreateTablet for tablet cafedad069bc481fb9f697a2c65dcfec (DEFAULT_TABLE table=test-table [id=a9e090eae25343489e420e5cc8fe95c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:44.363893 28377 tablet_service.cc:1467] Processing CreateTablet for tablet cafedad069bc481fb9f697a2c65dcfec (DEFAULT_TABLE table=test-table [id=a9e090eae25343489e420e5cc8fe95c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:44.364166 28656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafedad069bc481fb9f697a2c65dcfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:44.364213 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafedad069bc481fb9f697a2c65dcfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:44.364231 28903 tablet_service.cc:1467] Processing CreateTablet for tablet cafedad069bc481fb9f697a2c65dcfec (DEFAULT_TABLE table=test-table [id=a9e090eae25343489e420e5cc8fe95c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:44.364524 28903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cafedad069bc481fb9f697a2c65dcfec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:44.366708 29087 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: Bootstrap starting.
I20260810 15:19:44.367362 29087 tablet_bootstrap.cc:654] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:44.367420 29089 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Bootstrap starting.
I20260810 15:19:44.367671 29090 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Bootstrap starting.
I20260810 15:19:44.368023 29087 log.cc:824] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:44.368073 29089 tablet_bootstrap.cc:654] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:44.368271 29090 tablet_bootstrap.cc:654] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:44.368579 29089 log.cc:824] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:44.368877 29087 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: No bootstrap required, opened a new log
I20260810 15:19:44.368854 29090 log.cc:824] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:44.368942 29087 ts_tablet_manager.cc:1397] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:19:44.369326 29089 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: No bootstrap required, opened a new log
I20260810 15:19:44.369441 29089 ts_tablet_manager.cc:1397] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:19:44.369465 29090 tablet_bootstrap.cc:492] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: No bootstrap required, opened a new log
I20260810 15:19:44.369508 29090 ts_tablet_manager.cc:1397] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:44.370784 29090 raft_consensus.cc:348] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.370919 29090 raft_consensus.cc:374] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:44.370954 29090 raft_consensus.cc:729] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb44b5958504016823b527dcaf3cb26, State: Initialized, Role: FOLLOWER
I20260810 15:19:44.370857 29087 raft_consensus.cc:348] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.371017 29087 raft_consensus.cc:374] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:44.371052 29087 raft_consensus.cc:729] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3f8be114404b479c2ae870b18ad5e7, State: Initialized, Role: FOLLOWER
I20260810 15:19:44.371089 29090 consensus_queue.cc:260] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.371192 29087 consensus_queue.cc:260] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.371397 29090 ts_tablet_manager.cc:1428] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:19:44.371469 29087 ts_tablet_manager.cc:1428] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 15:19:44.371522 29049 heartbeater.cc:503] Master 127.29.230.62:40377 was elected leader, sending a full tablet report...
I20260810 15:19:44.371609 28499 heartbeater.cc:503] Master 127.29.230.62:40377 was elected leader, sending a full tablet report...
I20260810 15:19:44.371424 29089 raft_consensus.cc:348] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.371821 29089 raft_consensus.cc:374] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:44.371870 29089 raft_consensus.cc:729] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c5ac08892144fb9f811e2c69e40dd3, State: Initialized, Role: FOLLOWER
I20260810 15:19:44.371971 29089 consensus_queue.cc:260] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.372262 28784 heartbeater.cc:503] Master 127.29.230.62:40377 was elected leader, sending a full tablet report...
I20260810 15:19:44.372253 29089 ts_tablet_manager.cc:1428] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260810 15:19:44.407610 28785 tablet.cc:2369] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:44.447638 29095 raft_consensus.cc:482] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:44.447770 29095 raft_consensus.cc:504] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.448133 29095 leader_election.cc:288] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79c5ac08892144fb9f811e2c69e40dd3 (127.29.230.2:38593), 1b3f8be114404b479c2ae870b18ad5e7 (127.29.230.1:42799)
I20260810 15:19:44.451896 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:42163 (local address 127.29.230.2:38593)
0810 15:19:44.448472 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.448474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.448485 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:19:44.448516 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.448517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.448527 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.448584 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:44.448769 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.449558 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451112 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.451548 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451618 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.451628 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.451783 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.451824 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:19:44.451889 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.448326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:38593 (local address 127.29.230.3:42163)
0810 15:19:44.448387 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.448399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.448441 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.448614 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.448615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.448617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:44.448690 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.448692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.449667 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.449668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.451001 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.451005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.451630 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.451714 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.451717 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.451764 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.451808 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":50}
I20260810 15:19:44.451954 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.448422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:60283 (local address 127.29.230.1:42799)
0810 15:19:44.448462 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.448464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.448624 (+ 160us) server_negotiation.cc:407] Connection header received
0810 15:19:44.448657 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.448658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.448667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.448707 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:44.448844 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.449560 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451132 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.451563 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451636 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.451645 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.451846 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.451890 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:19:44.452170 29100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.448346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:42799 (local address 127.29.230.3:60283)
0810 15:19:44.448551 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.448558 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.448608 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.448726 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.448727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.448728 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:44.448783 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.448784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.449666 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.449668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.451001 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.451005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.451654 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.451655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.451772 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.451776 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.451827 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.451875 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":61,"thread_start_us":154,"threads_started":1}
I20260810 15:19:44.452263 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafedad069bc481fb9f697a2c65dcfec" candidate_uuid: "7fb44b5958504016823b527dcaf3cb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c5ac08892144fb9f811e2c69e40dd3" is_pre_election: true
I20260810 15:19:44.452438 28682 raft_consensus.cc:2454] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb44b5958504016823b527dcaf3cb26 in term 0.
I20260810 15:19:44.452523 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafedad069bc481fb9f697a2c65dcfec" candidate_uuid: "7fb44b5958504016823b527dcaf3cb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3f8be114404b479c2ae870b18ad5e7" is_pre_election: true
I20260810 15:19:44.452663 28831 leader_election.cc:302] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79c5ac08892144fb9f811e2c69e40dd3, 7fb44b5958504016823b527dcaf3cb26; no voters:
I20260810 15:19:44.452694 28411 raft_consensus.cc:2454] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb44b5958504016823b527dcaf3cb26 in term 0.
I20260810 15:19:44.452816 29095 raft_consensus.cc:2781] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:44.452876 29095 raft_consensus.cc:482] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:44.452898 29095 raft_consensus.cc:3037] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:44.453588 29095 raft_consensus.cc:504] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.453704 29095 leader_election.cc:288] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [CANDIDATE]: Term 1 election: Requested vote from peers 79c5ac08892144fb9f811e2c69e40dd3 (127.29.230.2:38593), 1b3f8be114404b479c2ae870b18ad5e7 (127.29.230.1:42799)
I20260810 15:19:44.453863 28682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafedad069bc481fb9f697a2c65dcfec" candidate_uuid: "7fb44b5958504016823b527dcaf3cb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79c5ac08892144fb9f811e2c69e40dd3"
I20260810 15:19:44.453858 28411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cafedad069bc481fb9f697a2c65dcfec" candidate_uuid: "7fb44b5958504016823b527dcaf3cb26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3f8be114404b479c2ae870b18ad5e7"
I20260810 15:19:44.453953 28682 raft_consensus.cc:3037] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:44.453964 28411 raft_consensus.cc:3037] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:44.454852 28682 raft_consensus.cc:2454] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb44b5958504016823b527dcaf3cb26 in term 1.
I20260810 15:19:44.454852 28411 raft_consensus.cc:2454] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb44b5958504016823b527dcaf3cb26 in term 1.
I20260810 15:19:44.455017 28831 leader_election.cc:302] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79c5ac08892144fb9f811e2c69e40dd3, 7fb44b5958504016823b527dcaf3cb26; no voters:
I20260810 15:19:44.455123 29095 raft_consensus.cc:2781] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:44.455286 29095 raft_consensus.cc:686] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 1 LEADER]: Becoming Leader. State: Replica: 7fb44b5958504016823b527dcaf3cb26, State: Running, Role: LEADER
I20260810 15:19:44.455379 29095 consensus_queue.cc:237] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } }
I20260810 15:19:44.456183 28143 catalog_manager.cc:5495] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb44b5958504016823b527dcaf3cb26 (127.29.230.3). New cstate: current_term: 1 leader_uuid: "7fb44b5958504016823b527dcaf3cb26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb44b5958504016823b527dcaf3cb26" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 } health_report { overall_health: UNKNOWN } } }
I20260810 15:19:44.478344 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.476235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.29.230.3:43489)
0810 15:19:44.476286 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.476288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.476298 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:44.476352 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.476353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.476369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.476423 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:44.476541 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.477033 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.477735 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.477918 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.477965 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.478082 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:44.478182 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:44.478238 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.478251 (+ 13us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.478267 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:19:44.480073 28682 raft_consensus.cc:1261] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 1 FOLLOWER]: Refusing update from remote peer 7fb44b5958504016823b527dcaf3cb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:44.480084 28411 raft_consensus.cc:1261] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7 [term 1 FOLLOWER]: Refusing update from remote peer 7fb44b5958504016823b527dcaf3cb26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:44.480414 29095 consensus_queue.cc:1035] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3f8be114404b479c2ae870b18ad5e7" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:44.480564 29103 consensus_queue.cc:1035] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79c5ac08892144fb9f811e2c69e40dd3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:44.482961 29108 mvcc.cc:205] Tried to move back new op lower bound from 7316992755627802624 to 7316992755529854976. Current Snapshot: MvccSnapshot[applied={T|T < 7316992755627802624}]
I20260810 15:19:44.484207 29111 mvcc.cc:205] Tried to move back new op lower bound from 7316992755627802624 to 7316992755529854976. Current Snapshot: MvccSnapshot[applied={T|T < 7316992755627802624 or (T in {7316992755627802624})}]
I20260810 15:19:44.484968 28144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:44.485065 28144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:44.485773 28144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90f81fe675994191a3401aa15d4d8864: Sending DeleteTablet for 3 replicas of tablet cafedad069bc481fb9f697a2c65dcfec
I20260810 15:19:44.486115 28656 tablet_service.cc:1514] Processing DeleteTablet for tablet cafedad069bc481fb9f697a2c65dcfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:44 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:36508
I20260810 15:19:44.486116 28903 tablet_service.cc:1514] Processing DeleteTablet for tablet cafedad069bc481fb9f697a2c65dcfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:44 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:44782
I20260810 15:19:44.486343 28377 tablet_service.cc:1514] Processing DeleteTablet for tablet cafedad069bc481fb9f697a2c65dcfec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:44 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:57460
I20260810 15:19:44.486456 29124 tablet_replica.cc:330] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: stopping tablet replica
I20260810 15:19:44.486569 29124 raft_consensus.cc:2229] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:44.486579 29127 tablet_replica.cc:330] T cafedad069bc481fb9f697a2c65dcfec P 1b3f8be114404b479c2ae870b18ad5e7: stopping tablet replica
I20260810 15:19:44.486665 29124 raft_consensus.cc:2258] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:44.486726 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 28250
I20260810 15:19:44.487100 29124 ts_tablet_manager.cc:1905] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:44.488935 29124 ts_tablet_manager.cc:1918] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:44.489001 29124 log.cc:1196] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/cafedad069bc481fb9f697a2c65dcfec
I20260810 15:19:44.489300 29124 ts_tablet_manager.cc:1939] T cafedad069bc481fb9f697a2c65dcfec P 7fb44b5958504016823b527dcaf3cb26: Deleting consensus metadata
I20260810 15:19:44.489730 28116 catalog_manager.cc:4841] TS 7fb44b5958504016823b527dcaf3cb26 (127.29.230.3:43489): tablet cafedad069bc481fb9f697a2c65dcfec (table test-table [id=a9e090eae25343489e420e5cc8fe95c9]) successfully deleted
I20260810 15:19:44.502961 29125 tablet_replica.cc:330] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: stopping tablet replica
I20260810 15:19:44.503109 29125 raft_consensus.cc:2229] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:44.503237 29125 raft_consensus.cc:2258] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:44.504715 29125 ts_tablet_manager.cc:1905] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:44.506142 29125 ts_tablet_manager.cc:1918] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:44.506212 29125 log.cc:1196] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/cafedad069bc481fb9f697a2c65dcfec
I20260810 15:19:44.506449 29125 ts_tablet_manager.cc:1939] T cafedad069bc481fb9f697a2c65dcfec P 79c5ac08892144fb9f811e2c69e40dd3: Deleting consensus metadata
I20260810 15:19:44.506886 28125 catalog_manager.cc:4841] TS 79c5ac08892144fb9f811e2c69e40dd3 (127.29.230.2:38593): tablet cafedad069bc481fb9f697a2c65dcfec (table test-table [id=a9e090eae25343489e420e5cc8fe95c9]) successfully deleted
W20260810 15:19:44.509618 28118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:42799 (error 108)
I20260810 15:19:44.509869 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 28508
I20260810 15:19:44.515990 29067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.509999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:42799 (local address 127.0.0.1:57476)
0810 15:19:44.515878 (+ 5879us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.515946 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:42799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5845}
W20260810 15:19:44.519033 28118 catalog_manager.cc:4584] TS 1b3f8be114404b479c2ae870b18ad5e7 (127.29.230.1:42799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cafedad069bc481fb9f697a2c65dcfec: Network error: Client connection negotiation failed: client connection to 127.29.230.1:42799: connect: Connection refused (error 111)
I20260810 15:19:44.527562 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 28797
I20260810 15:19:44.544117 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 28077
2026-08-10T15:19:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T15:19:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:44Z Disabled control of system clock
I20260810 15:19:44.578313 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35887
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:40817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:44.658648 29136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:44.658850 29136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:44.658869 29136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:44.660022 29136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:44.660064 29136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:44.660075 29136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:44.660086 29136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:44.660099 29136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:44.661432 29136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:35887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:44.661662 29136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:44.661960 29136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:44.666626 29155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.666618 29152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.666620 29148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.666877 29136 server_base.cc:1034] running on GCE node
I20260810 15:19:44.667107 29136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:44.667426 29136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:44.668576 29136 hybrid_clock.cc:648] HybridClock initialized: now 1786375184668561 us; error 29 us; skew 500 ppm
I20260810 15:19:44.669744 29136 webserver.cc:460] Webserver started at http://127.29.230.62:39901/ using document root <none> and password file <none>
I20260810 15:19:44.669981 29136 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:44.670032 29136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:44.670163 29136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:44.671187 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "85bc9f66f3bb469fb6e61bb161bca034"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7cbe208a16c9c3143409a3dc5eb005cd"
server_key_iv: "8bf49c9b9cd976b855de4a89501a3f87"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.671577 29136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "85bc9f66f3bb469fb6e61bb161bca034"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7cbe208a16c9c3143409a3dc5eb005cd"
server_key_iv: "8bf49c9b9cd976b855de4a89501a3f87"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.672848 29136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:44.673462 29165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.673612 29136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:44.673693 29136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "85bc9f66f3bb469fb6e61bb161bca034"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7cbe208a16c9c3143409a3dc5eb005cd"
server_key_iv: "8bf49c9b9cd976b855de4a89501a3f87"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.673759 29136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:44.683028 29136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.683336 29136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.683459 29136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.687129 29136 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:35887
I20260810 15:19:44.687189 29274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:35887 every 8 connection(s)
I20260810 15:19:44.687529 29136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:44.688500 29276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:44.690943 29276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: Bootstrap starting.
I20260810 15:19:44.691572 29276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:44.691843 29276 log.cc:824] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:44.692548 29276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: No bootstrap required, opened a new log
I20260810 15:19:44.692593 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 29136
I20260810 15:19:44.692685 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:44.692875 30616 external_mini_cluster.cc:1428] Setting key 56940aa03ce3e93e1e2389f6749a2fe7
I20260810 15:19:44.694571 29276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } }
I20260810 15:19:44.694739 29276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:44.694777 29276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85bc9f66f3bb469fb6e61bb161bca034, State: Initialized, Role: FOLLOWER
I20260810 15:19:44.694917 29276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } }
I20260810 15:19:44.695041 29276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:44.695086 29276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:44.695133 29276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:44.695844 29276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } }
I20260810 15:19:44.695951 29276 leader_election.cc:302] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85bc9f66f3bb469fb6e61bb161bca034; no voters:
I20260810 15:19:44.696116 29276 leader_election.cc:288] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:44.696161 29284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:44.696313 29284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [term 1 LEADER]: Becoming Leader. State: Replica: 85bc9f66f3bb469fb6e61bb161bca034, State: Running, Role: LEADER
I20260810 15:19:44.696358 29276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:44.696408 29284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } }
I20260810 15:19:44.696758 29284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85bc9f66f3bb469fb6e61bb161bca034. Latest consensus state: current_term: 1 leader_uuid: "85bc9f66f3bb469fb6e61bb161bca034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } } }
I20260810 15:19:44.696851 29284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:44.696767 29291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85bc9f66f3bb469fb6e61bb161bca034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85bc9f66f3bb469fb6e61bb161bca034" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 35887 } } }
I20260810 15:19:44.696936 29291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034 [sys.catalog]: This master's current role is: LEADER
W20260810 15:19:44.698096 29321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:19:44.698153 29321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:19:44.698205 29303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:44.698351 29303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:44.699759 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.693398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.29.230.62:35887)
0810 15:19:44.693604 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.693613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.693647 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:19:44.694534 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.694543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.694621 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.694766 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.694977 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.696681 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.697782 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.698281 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.698401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.699047 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.699051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.699051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:44.699242 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.699308 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:44.699323 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.699397 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.699477 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.699557 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260810 15:19:44.700139 29303 catalog_manager.cc:1347] Generated new cluster ID: d9e4c493251348c0bf0fe0248cb87ebb
I20260810 15:19:44.700194 29303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:44.714356 29303 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:44.715068 29303 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:44.718853 29303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: Generated new TSK 0
I20260810 15:19:44.719070 29303 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:44.723244 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--builtin_ntp_servers=127.29.230.20:40817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:44.805078 29328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:44.805279 29328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:44.805310 29328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:44.806438 29328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:44.806476 29328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:44.806509 29328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:44.807906 29328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:44.808145 29328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:44.808445 29328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:44.814432 29339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.814414 29340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.814678 29328 server_base.cc:1034] running on GCE node
W20260810 15:19:44.814414 29345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.814939 29328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:44.815168 29328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:44.816313 29328 hybrid_clock.cc:648] HybridClock initialized: now 1786375184816313 us; error 44 us; skew 500 ppm
I20260810 15:19:44.817564 29328 webserver.cc:460] Webserver started at http://127.29.230.1:33029/ using document root <none> and password file <none>
I20260810 15:19:44.817823 29328 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:44.817873 29328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:44.818003 29328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:44.819069 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "dff7235afe824dad9856c35eb7aabf6f"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "ee1fbccb4485bbc2aceecb02cbba4fa6"
server_key_iv: "1731d1eb027f30566c52c47dd1cc09c0"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.819435 29328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "dff7235afe824dad9856c35eb7aabf6f"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "ee1fbccb4485bbc2aceecb02cbba4fa6"
server_key_iv: "1731d1eb027f30566c52c47dd1cc09c0"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.820689 29328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:19:44.821412 29357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.821552 29328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:19:44.821648 29328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "dff7235afe824dad9856c35eb7aabf6f"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "ee1fbccb4485bbc2aceecb02cbba4fa6"
server_key_iv: "1731d1eb027f30566c52c47dd1cc09c0"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.821707 29328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:44.841344 29328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.841678 29328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.841791 29328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.842013 29328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:44.842367 29328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:44.842404 29328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.842425 29328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:44.842437 29328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.848146 29328 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:38231
I20260810 15:19:44.848234 29576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:38231 every 8 connection(s)
I20260810 15:19:44.848563 29328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:44.849334 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 29328
I20260810 15:19:44.849438 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:44.849654 30616 external_mini_cluster.cc:1428] Setting key c43596e16eaf91e886c4e128e190658c
I20260810 15:19:44.850669 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--builtin_ntp_servers=127.29.230.20:40817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:19:44.854801 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.849368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:44481 (local address 127.29.230.62:35887)
0810 15:19:44.849469 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.849471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.849622 (+ 151us) server_negotiation.cc:407] Connection header received
0810 15:19:44.850235 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.850237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.850252 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.850327 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.851087 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.851798 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.853225 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.853427 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.853496 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.854440 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.854442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.854442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:44.854450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.854468 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:44.854472 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.854529 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.854683 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.854717 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 15:19:44.855032 29586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.849284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35887 (local address 127.29.230.1:44481)
0810 15:19:44.849484 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.849527 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.850131 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.850406 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.850411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.850422 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:44.850997 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.851003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.851899 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.851900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.853128 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.853132 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.853510 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.853511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.853641 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.854286 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:44.854286 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:44.854317 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:44.854343 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:44.854582 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:44.854590 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:44.854593 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.854665 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.854762 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260810 15:19:44.855581 29581 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35887
I20260810 15:19:44.855677 29581 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:44.855865 29581 heartbeater.cc:511] Master 127.29.230.62:35887 requested a full tablet report, sending...
I20260810 15:19:44.856374 29196 ts_manager.cc:194] Registered new tserver with Master: dff7235afe824dad9856c35eb7aabf6f (127.29.230.1:38231)
I20260810 15:19:44.856982 29196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.1:44481
W20260810 15:19:44.934548 29588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:44.934748 29588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:44.934767 29588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:44.935959 29588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:44.936002 29588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:44.936028 29588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:44.937390 29588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:44.937582 29588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:44.937842 29588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:44.943076 29598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.943084 29597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:44.943085 29603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:44.943395 29588 server_base.cc:1034] running on GCE node
I20260810 15:19:44.943598 29588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:44.943802 29588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:44.944931 29588 hybrid_clock.cc:648] HybridClock initialized: now 1786375184944914 us; error 33 us; skew 500 ppm
I20260810 15:19:44.946250 29588 webserver.cc:460] Webserver started at http://127.29.230.2:38703/ using document root <none> and password file <none>
I20260810 15:19:44.946462 29588 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:44.946506 29588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:44.946624 29588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:44.947734 29588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "b55206b69c714786a4eaa067cda39be8"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7fad195d78fb69c16b7ad5c60656a09f"
server_key_iv: "6afbcb156e57ffa2a8b652bceabc5be3"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.948067 29588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "b55206b69c714786a4eaa067cda39be8"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7fad195d78fb69c16b7ad5c60656a09f"
server_key_iv: "6afbcb156e57ffa2a8b652bceabc5be3"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.949339 29588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:44.950071 29620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.950229 29588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:44.950311 29588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "b55206b69c714786a4eaa067cda39be8"
format_stamp: "Formatted at 2026-08-10 15:19:44 on dist-test-slave-gjrz"
server_key: "7fad195d78fb69c16b7ad5c60656a09f"
server_key_iv: "6afbcb156e57ffa2a8b652bceabc5be3"
server_key_version: "encryptionkey@0"
I20260810 15:19:44.950380 29588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:44.973498 29588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:44.973812 29588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:44.973945 29588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:44.974196 29588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:44.974539 29588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:44.974581 29588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.974606 29588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:44.974628 29588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:44.980223 29588 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42631
I20260810 15:19:44.980283 29836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42631 every 8 connection(s)
I20260810 15:19:44.980587 29588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:44.986083 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.981259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:55715 (local address 127.29.230.62:35887)
0810 15:19:44.981303 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:44.981305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:44.981532 (+ 227us) server_negotiation.cc:407] Connection header received
0810 15:19:44.982029 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:44.982030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:44.982043 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:44.982106 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:44.982729 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.983239 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.984359 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.984671 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.984748 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.985776 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:44.985778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:44.985778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:44.985784 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:44.985803 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:44.985808 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:44.985869 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:44.985966 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 15:19:44.986013 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:19:44.986294 29840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:44.981254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35887 (local address 127.29.230.2:55715)
0810 15:19:44.981414 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:44.981449 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:44.981937 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:44.982163 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:44.982166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:44.982174 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:44.982664 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.982668 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.983330 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.983331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.984267 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:44.984270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:44.984756 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:44.984758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:44.984916 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:44.985528 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:44.985528 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:44.985559 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:44.985591 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:44.985872 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:44.985878 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:44.985882 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:44.985964 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 15:19:44.986076 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260810 15:19:44.986902 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 29588
I20260810 15:19:44.987061 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:44.987185 29837 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35887
I20260810 15:19:44.987298 30616 external_mini_cluster.cc:1428] Setting key 5587337752d143eb4150ffec2c7c8ab5
I20260810 15:19:44.987304 29837 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:44.987505 29837 heartbeater.cc:511] Master 127.29.230.62:35887 requested a full tablet report, sending...
I20260810 15:19:44.988107 29196 ts_manager.cc:194] Registered new tserver with Master: b55206b69c714786a4eaa067cda39be8 (127.29.230.2:42631)
I20260810 15:19:44.988224 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--builtin_ntp_servers=127.29.230.20:40817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:19:44.988538 29196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.2:55715
W20260810 15:19:45.069193 29841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:45.069387 29841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:45.069408 29841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:45.070499 29841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:45.070528 29841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:45.070550 29841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:45.071841 29841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:35887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:45.072041 29841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:45.072285 29841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:45.077363 29858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:45.077371 29863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:45.077389 29849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:45.077682 29841 server_base.cc:1034] running on GCE node
I20260810 15:19:45.077992 29841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:45.078282 29841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:45.079424 29841 hybrid_clock.cc:648] HybridClock initialized: now 1786375185079408 us; error 28 us; skew 500 ppm
I20260810 15:19:45.080688 29841 webserver.cc:460] Webserver started at http://127.29.230.3:35075/ using document root <none> and password file <none>
I20260810 15:19:45.080919 29841 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:45.080969 29841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:45.081105 29841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:45.082187 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "08b7fc853b584457af716bc413c0e0fc"
format_stamp: "Formatted at 2026-08-10 15:19:45 on dist-test-slave-gjrz"
server_key: "57188546104aeee00d97c965044d7f6b"
server_key_iv: "043ca46699b1c6144c301a352ded04d9"
server_key_version: "encryptionkey@0"
I20260810 15:19:45.082517 29841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "08b7fc853b584457af716bc413c0e0fc"
format_stamp: "Formatted at 2026-08-10 15:19:45 on dist-test-slave-gjrz"
server_key: "57188546104aeee00d97c965044d7f6b"
server_key_iv: "043ca46699b1c6144c301a352ded04d9"
server_key_version: "encryptionkey@0"
I20260810 15:19:45.083886 29841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:45.084671 29884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:45.084846 29841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:45.084931 29841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "08b7fc853b584457af716bc413c0e0fc"
format_stamp: "Formatted at 2026-08-10 15:19:45 on dist-test-slave-gjrz"
server_key: "57188546104aeee00d97c965044d7f6b"
server_key_iv: "043ca46699b1c6144c301a352ded04d9"
server_key_version: "encryptionkey@0"
I20260810 15:19:45.084992 29841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:45.108246 29841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:45.108565 29841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:45.108698 29841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:45.108922 29841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:45.109261 29841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:45.109295 29841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:45.109324 29841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:45.109341 29841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:45.114799 29841 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:43687
I20260810 15:19:45.114857 30103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:43687 every 8 connection(s)
I20260810 15:19:45.115188 29841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:45.120368 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.115911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:56609 (local address 127.29.230.62:35887)
0810 15:19:45.115971 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.115973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.116231 (+ 258us) server_negotiation.cc:407] Connection header received
0810 15:19:45.116751 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.116752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.116766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.116816 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:45.117526 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.118045 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.119015 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.119215 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.119291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.120093 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:45.120095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:45.120095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:45.120100 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:45.120117 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:45.120122 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:45.120166 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.120250 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.120287 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:19:45.120551 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.115882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:35887 (local address 127.29.230.3:56609)
0810 15:19:45.116104 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.116139 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.116635 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.116901 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.116904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.116913 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:45.117421 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.117426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.118140 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.118141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.118923 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.118925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.119307 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.119308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.119439 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.119938 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:45.119939 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:45.119970 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:45.120003 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:45.120174 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:45.120179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:45.120182 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.120245 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.120348 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260810 15:19:45.121013 30105 heartbeater.cc:347] Connected to a master server at 127.29.230.62:35887
I20260810 15:19:45.121112 30105 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:45.121289 30105 heartbeater.cc:511] Master 127.29.230.62:35887 requested a full tablet report, sending...
I20260810 15:19:45.121626 29196 ts_manager.cc:194] Registered new tserver with Master: 08b7fc853b584457af716bc413c0e0fc (127.29.230.3:43687)
I20260810 15:19:45.121971 29196 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.3:56609
I20260810 15:19:45.124084 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 29841
I20260810 15:19:45.124168 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:45.124362 30616 external_mini_cluster.cc:1428] Setting key 7d32af6c3a60c4ca27bde34f2e675541
I20260810 15:19:45.125386 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:45.129500 29282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.126729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.29.230.62:35887)
0810 15:19:45.126769 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.126771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.126896 (+ 125us) server_negotiation.cc:407] Connection header received
0810 15:19:45.126935 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.126936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.126950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.127072 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:45.127207 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.127723 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.128498 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.128681 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.128733 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.129297 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:45.129298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:45.129299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:45.129303 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:45.129316 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:45.129319 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:45.129358 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.129403 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.129425 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:19:45.131038 29196 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:45.132560 29196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:45.138784 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.135303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.29.230.3:43687)
0810 15:19:45.135436 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.135440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.135450 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:45.135482 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.135483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.135511 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.135591 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:45.135796 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.136513 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.137810 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.138394 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.138465 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.138565 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.138656 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.138696 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260810 15:19:45.138770 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.135186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:43687 (local address 127.0.0.1:43132)
0810 15:19:45.135311 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.135319 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.135360 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.135655 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.135656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.135658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:45.135720 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.135722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.136594 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.136595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.137737 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.137739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.138472 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.138473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.138555 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.138568 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.138640 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.138695 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":24,"thread_start_us":67,"threads_started":1}
I20260810 15:19:45.139117 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.134748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:38231 (local address 127.0.0.1:49392)
0810 15:19:45.134881 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.134912 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.134975 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.135560 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.135564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.135567 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:45.135638 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.135641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.137051 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.137052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.138009 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.138011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.138908 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.138909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.138982 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.138987 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.139014 (+ 27us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.139051 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260810 15:19:45.139513 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.134881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.29.230.1:38231)
0810 15:19:45.135382 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.135387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.135398 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:19:45.135441 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.135443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.135459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.135758 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:45.135827 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.136959 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.138096 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.138801 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.139227 (+ 426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.139356 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.139382 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.139424 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":90,"threads_started":1}
I20260810 15:19:45.139604 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.135014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42631 (local address 127.0.0.1:38190)
0810 15:19:45.135131 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.135141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.135187 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.135644 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.135645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.135647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:45.135718 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.135720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.137078 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.137079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.137960 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.137963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.139364 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.139366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.139434 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.139437 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.139501 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.139540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260810 15:19:45.139585 29968 tablet_service.cc:1467] Processing CreateTablet for tablet f02614abfb3649879fc854c378a2b36c (DEFAULT_TABLE table=test-table [id=8034ff4b709b40c5a54ea3b8f074de41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:45.139886 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.135142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.29.230.2:42631)
0810 15:19:45.135486 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.135490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.135499 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:45.135534 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.135536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.135551 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.135794 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:45.135856 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.136980 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.138045 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.138802 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.139360 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.139447 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.139513 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.139802 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260810 15:19:45.140007 29968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02614abfb3649879fc854c378a2b36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:45.140214 29455 tablet_service.cc:1467] Processing CreateTablet for tablet f02614abfb3649879fc854c378a2b36c (DEFAULT_TABLE table=test-table [id=8034ff4b709b40c5a54ea3b8f074de41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:45.140573 29455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02614abfb3649879fc854c378a2b36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:45.140486 29722 tablet_service.cc:1467] Processing CreateTablet for tablet f02614abfb3649879fc854c378a2b36c (DEFAULT_TABLE table=test-table [id=8034ff4b709b40c5a54ea3b8f074de41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:45.140794 29722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02614abfb3649879fc854c378a2b36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:45.143347 30132 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Bootstrap starting.
I20260810 15:19:45.143877 30135 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Bootstrap starting.
I20260810 15:19:45.143904 30134 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Bootstrap starting.
I20260810 15:19:45.144016 30132 tablet_bootstrap.cc:654] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:45.144369 30132 log.cc:824] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:45.144491 30134 tablet_bootstrap.cc:654] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:45.144712 30135 tablet_bootstrap.cc:654] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:45.144776 30134 log.cc:824] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:45.145061 30135 log.cc:824] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:45.145180 30132 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: No bootstrap required, opened a new log
I20260810 15:19:45.145252 30132 ts_tablet_manager.cc:1397] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:45.145453 30134 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: No bootstrap required, opened a new log
I20260810 15:19:45.145514 30134 ts_tablet_manager.cc:1397] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:45.147012 30132 raft_consensus.cc:348] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.147156 30132 raft_consensus.cc:374] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:45.147177 30132 raft_consensus.cc:729] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55206b69c714786a4eaa067cda39be8, State: Initialized, Role: FOLLOWER
I20260810 15:19:45.147295 30132 consensus_queue.cc:260] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.147553 30132 ts_tablet_manager.cc:1428] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:19:45.147643 29837 heartbeater.cc:503] Master 127.29.230.62:35887 was elected leader, sending a full tablet report...
I20260810 15:19:45.147549 30134 raft_consensus.cc:348] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.147692 30134 raft_consensus.cc:374] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:45.147722 30134 raft_consensus.cc:729] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b7fc853b584457af716bc413c0e0fc, State: Initialized, Role: FOLLOWER
I20260810 15:19:45.147830 30134 consensus_queue.cc:260] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.148105 30134 ts_tablet_manager.cc:1428] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:45.148113 30135 tablet_bootstrap.cc:492] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: No bootstrap required, opened a new log
I20260810 15:19:45.148154 30105 heartbeater.cc:503] Master 127.29.230.62:35887 was elected leader, sending a full tablet report...
I20260810 15:19:45.148176 30135 ts_tablet_manager.cc:1397] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:19:45.150122 30135 raft_consensus.cc:348] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.150270 30135 raft_consensus.cc:374] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:45.150315 30135 raft_consensus.cc:729] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff7235afe824dad9856c35eb7aabf6f, State: Initialized, Role: FOLLOWER
I20260810 15:19:45.150414 30135 consensus_queue.cc:260] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.150663 30135 ts_tablet_manager.cc:1428] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:19:45.150712 29581 heartbeater.cc:503] Master 127.29.230.62:35887 was elected leader, sending a full tablet report...
W20260810 15:19:45.231137 29839 tablet.cc:2369] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:19:45.349263 29583 tablet.cc:2369] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:45.357165 30147 raft_consensus.cc:482] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:45.357277 30147 raft_consensus.cc:504] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.357643 30147 leader_election.cc:288] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b55206b69c714786a4eaa067cda39be8 (127.29.230.2:42631), 08b7fc853b584457af716bc413c0e0fc (127.29.230.3:43687)
I20260810 15:19:45.361578 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.357907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:39685 (local address 127.29.230.2:42631)
0810 15:19:45.357969 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.357972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.358112 (+ 140us) server_negotiation.cc:407] Connection header received
0810 15:19:45.358149 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.358151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.358162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.358223 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:45.358395 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.359145 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.360645 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.361207 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.361284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.361292 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.361454 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.361491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:19:45.361567 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.358134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:39243 (local address 127.29.230.3:43687)
0810 15:19:45.358193 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.358195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.358203 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:19:45.358266 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.358267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.358275 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.358322 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:45.358439 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.359149 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.360646 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.361219 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.361284 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.361292 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.361454 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.361491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:19:45.361604 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.357845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42631 (local address 127.29.230.1:39685)
0810 15:19:45.357982 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.357993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.358072 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.358254 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.358255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.358258 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:45.358333 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.358335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.359223 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.359224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.360541 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.360545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.361294 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.361295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.361379 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.361387 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.361444 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.361499 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":86,"thread_start_us":81,"threads_started":1}
I20260810 15:19:45.361608 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.358041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:43687 (local address 127.29.230.1:39243)
0810 15:19:45.358157 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.358165 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:45.358199 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:45.358331 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:45.358332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:45.358333 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:45.358385 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.358387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.359221 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.359221 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.360541 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:45.360545 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.361292 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.361294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:45.361379 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.361386 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 15:19:45.361444 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:19:45.361499 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":23,"thread_start_us":84,"threads_started":1}
I20260810 15:19:45.362016 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02614abfb3649879fc854c378a2b36c" candidate_uuid: "dff7235afe824dad9856c35eb7aabf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b7fc853b584457af716bc413c0e0fc" is_pre_election: true
I20260810 15:19:45.362010 29759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02614abfb3649879fc854c378a2b36c" candidate_uuid: "dff7235afe824dad9856c35eb7aabf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55206b69c714786a4eaa067cda39be8" is_pre_election: true
I20260810 15:19:45.362169 29759 raft_consensus.cc:2454] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff7235afe824dad9856c35eb7aabf6f in term 0.
I20260810 15:19:45.362169 30013 raft_consensus.cc:2454] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff7235afe824dad9856c35eb7aabf6f in term 0.
I20260810 15:19:45.362385 29358 leader_election.cc:302] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b55206b69c714786a4eaa067cda39be8, dff7235afe824dad9856c35eb7aabf6f; no voters:
I20260810 15:19:45.362532 30147 raft_consensus.cc:2781] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:45.362601 30147 raft_consensus.cc:482] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:45.362623 30147 raft_consensus.cc:3037] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:45.363315 30147 raft_consensus.cc:504] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
I20260810 15:19:45.363428 30147 leader_election.cc:288] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [CANDIDATE]: Term 1 election: Requested vote from peers b55206b69c714786a4eaa067cda39be8 (127.29.230.2:42631), 08b7fc853b584457af716bc413c0e0fc (127.29.230.3:43687)
I20260810 15:19:45.363618 29759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02614abfb3649879fc854c378a2b36c" candidate_uuid: "dff7235afe824dad9856c35eb7aabf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b55206b69c714786a4eaa067cda39be8"
I20260810 15:19:45.363629 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02614abfb3649879fc854c378a2b36c" candidate_uuid: "dff7235afe824dad9856c35eb7aabf6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b7fc853b584457af716bc413c0e0fc"
I20260810 15:19:45.363699 30013 raft_consensus.cc:3037] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:45.363703 29759 raft_consensus.cc:3037] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:45.364583 30013 raft_consensus.cc:2454] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff7235afe824dad9856c35eb7aabf6f in term 1.
I20260810 15:19:45.364626 29759 raft_consensus.cc:2454] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff7235afe824dad9856c35eb7aabf6f in term 1.
I20260810 15:19:45.364777 29358 leader_election.cc:302] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b55206b69c714786a4eaa067cda39be8, dff7235afe824dad9856c35eb7aabf6f; no voters:
I20260810 15:19:45.364880 30147 raft_consensus.cc:2781] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:45.365041 30147 raft_consensus.cc:686] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 1 LEADER]: Becoming Leader. State: Replica: dff7235afe824dad9856c35eb7aabf6f, State: Running, Role: LEADER
I20260810 15:19:45.365130 30147 consensus_queue.cc:237] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } }
W20260810 15:19:45.365692 30107 tablet.cc:2369] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:45.366046 29196 catalog_manager.cc:5495] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f reported cstate change: term changed from 0 to 1, leader changed from <none> to dff7235afe824dad9856c35eb7aabf6f (127.29.230.1). New cstate: current_term: 1 leader_uuid: "dff7235afe824dad9856c35eb7aabf6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff7235afe824dad9856c35eb7aabf6f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 38231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 } health_report { overall_health: UNKNOWN } } }
I20260810 15:19:45.424757 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.422434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.29.230.1:38231)
0810 15:19:45.422498 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:45.422500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:45.422509 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:45.422565 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:45.422566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:45.422587 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:45.422647 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:45.422783 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.423256 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.424003 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:45.424219 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:45.424277 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:45.424449 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:45.424562 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:45.424637 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:45.424658 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 15:19:45.424680 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 15:19:45.426604 30013 raft_consensus.cc:1261] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 1 FOLLOWER]: Refusing update from remote peer dff7235afe824dad9856c35eb7aabf6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:45.426604 29759 raft_consensus.cc:1261] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 1 FOLLOWER]: Refusing update from remote peer dff7235afe824dad9856c35eb7aabf6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:45.426940 30147 consensus_queue.cc:1035] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b55206b69c714786a4eaa067cda39be8" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:45.427105 30155 consensus_queue.cc:1035] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7fc853b584457af716bc413c0e0fc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 43687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:45.429677 30163 mvcc.cc:205] Tried to move back new op lower bound from 7316992759504654336 to 7316992759256190976. Current Snapshot: MvccSnapshot[applied={T|T < 7316992759504654336}]
I20260810 15:19:45.431815 29196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:45.431913 29196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:45.432643 29196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85bc9f66f3bb469fb6e61bb161bca034: Sending DeleteTablet for 3 replicas of tablet f02614abfb3649879fc854c378a2b36c
I20260810 15:19:45.432940 29455 tablet_service.cc:1514] Processing DeleteTablet for tablet f02614abfb3649879fc854c378a2b36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:45 UTC) from {username='slave'} at 127.0.0.1:49392
I20260810 15:19:45.433005 29722 tablet_service.cc:1514] Processing DeleteTablet for tablet f02614abfb3649879fc854c378a2b36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:45 UTC) from {username='slave'} at 127.0.0.1:38190
I20260810 15:19:45.433049 29968 tablet_service.cc:1514] Processing DeleteTablet for tablet f02614abfb3649879fc854c378a2b36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:45 UTC) from {username='slave'} at 127.0.0.1:43132
I20260810 15:19:45.433238 30179 tablet_replica.cc:330] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f: stopping tablet replica
I20260810 15:19:45.433341 30179 raft_consensus.cc:2229] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:45.433393 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 29328
I20260810 15:19:45.433429 30179 raft_consensus.cc:2258] T f02614abfb3649879fc854c378a2b36c P dff7235afe824dad9856c35eb7aabf6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:45.433562 30180 tablet_replica.cc:330] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: stopping tablet replica
I20260810 15:19:45.433655 30180 raft_consensus.cc:2229] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:45.433741 30180 raft_consensus.cc:2258] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:45.439755 30180 ts_tablet_manager.cc:1905] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:45.440658 30181 tablet_replica.cc:330] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: stopping tablet replica
I20260810 15:19:45.440809 30181 raft_consensus.cc:2229] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:45.440989 30181 raft_consensus.cc:2258] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:45.441571 30180 ts_tablet_manager.cc:1918] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:45.441679 30180 log.cc:1196] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/f02614abfb3649879fc854c378a2b36c
I20260810 15:19:45.441948 30180 ts_tablet_manager.cc:1939] T f02614abfb3649879fc854c378a2b36c P b55206b69c714786a4eaa067cda39be8: Deleting consensus metadata
I20260810 15:19:45.442404 29166 catalog_manager.cc:4841] TS b55206b69c714786a4eaa067cda39be8 (127.29.230.2:42631): tablet f02614abfb3649879fc854c378a2b36c (table test-table [id=8034ff4b709b40c5a54ea3b8f074de41]) successfully deleted
I20260810 15:19:45.442521 30181 ts_tablet_manager.cc:1905] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:45.443799 30181 ts_tablet_manager.cc:1918] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:45.443871 30181 log.cc:1196] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/f02614abfb3649879fc854c378a2b36c
I20260810 15:19:45.444077 30181 ts_tablet_manager.cc:1939] T f02614abfb3649879fc854c378a2b36c P 08b7fc853b584457af716bc413c0e0fc: Deleting consensus metadata
I20260810 15:19:45.444437 29166 catalog_manager.cc:4841] TS 08b7fc853b584457af716bc413c0e0fc (127.29.230.3:43687): tablet f02614abfb3649879fc854c378a2b36c (table test-table [id=8034ff4b709b40c5a54ea3b8f074de41]) successfully deleted
W20260810 15:19:45.447623 29166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:38231 (error 108)
I20260810 15:19:45.448091 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:45.447963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:38231 (local address 127.0.0.1:49404)
0810 15:19:45.448015 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:45.448055 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:38231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 15:19:45.448170 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 29588
W20260810 15:19:45.448220 29166 catalog_manager.cc:4584] TS dff7235afe824dad9856c35eb7aabf6f (127.29.230.1:38231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f02614abfb3649879fc854c378a2b36c: Network error: Client connection negotiation failed: client connection to 127.29.230.1:38231: connect: Connection refused (error 111)
I20260810 15:19:45.457244 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 29841
I20260810 15:19:45.468022 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 29136
2026-08-10T15:19:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (922 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:45 dist-test-slave-gjrz krb5kdc[30191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:45 dist-test-slave-gjrz krb5kdc[30191](info): set up 2 sockets
Aug 10 15:19:45 dist-test-slave-gjrz krb5kdc[30191](info): commencing operation
krb5kdc: starting...
W20260810 15:19:46.651454 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:46 dist-test-slave-gjrz krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:19:46.770316 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33949
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:33949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:46.851680 30217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:46.851892 30217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:46.851922 30217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:46.853030 30217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:46.853061 30217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:46.853075 30217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:46.853085 30217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:46.853096 30217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:46.854547 30217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:33949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:46.854780 30217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:46.855028 30217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:19:46.859783 30217 server_base.cc:1034] running on GCE node
W20260810 15:19:46.859747 30228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:46.859753 30227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:46.859779 30233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:46.860162 30217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:46.860437 30217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:46.861593 30217 hybrid_clock.cc:648] HybridClock initialized: now 1786375186861578 us; error 27 us; skew 500 ppm
Aug 10 15:19:46 dist-test-slave-gjrz krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375186, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:46.864013 30217 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:19:46.864416 30217 webserver.cc:460] Webserver started at http://127.29.230.62:36559/ using document root <none> and password file <none>
I20260810 15:19:46.864626 30217 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:46.864671 30217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:46.864812 30217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:46.865864 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "999dec29b8504c69a3296d798ca1086d"
format_stamp: "Formatted at 2026-08-10 15:19:46 on dist-test-slave-gjrz"
server_key: "4ff2b3f3ff1572ee13e9bbee0cd4d5f0"
server_key_iv: "b26348eef552725e560c1424b023c304"
server_key_version: "encryptionkey@0"
I20260810 15:19:46.866194 30217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "999dec29b8504c69a3296d798ca1086d"
format_stamp: "Formatted at 2026-08-10 15:19:46 on dist-test-slave-gjrz"
server_key: "4ff2b3f3ff1572ee13e9bbee0cd4d5f0"
server_key_iv: "b26348eef552725e560c1424b023c304"
server_key_version: "encryptionkey@0"
I20260810 15:19:46.867458 30217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:46.868221 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:46.868387 30217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:46.868471 30217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "999dec29b8504c69a3296d798ca1086d"
format_stamp: "Formatted at 2026-08-10 15:19:46 on dist-test-slave-gjrz"
server_key: "4ff2b3f3ff1572ee13e9bbee0cd4d5f0"
server_key_iv: "b26348eef552725e560c1424b023c304"
server_key_version: "encryptionkey@0"
I20260810 15:19:46.868531 30217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:46.881624 30217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:46.883548 30217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:46.883674 30217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:46.887516 30217 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:33949
I20260810 15:19:46.887549 30334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:33949 every 8 connection(s)
I20260810 15:19:46.887866 30217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:46.888733 30335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:46.891031 30335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: Bootstrap starting.
I20260810 15:19:46.891641 30335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:46.891939 30335 log.cc:824] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:46.892619 30335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: No bootstrap required, opened a new log
I20260810 15:19:46.894181 30335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } }
I20260810 15:19:46.894318 30335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:46.894340 30335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999dec29b8504c69a3296d798ca1086d, State: Initialized, Role: FOLLOWER
I20260810 15:19:46.894482 30335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } }
I20260810 15:19:46.894546 30335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:46.894567 30335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:46.894613 30335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:46.895149 30335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } }
I20260810 15:19:46.895236 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30217
I20260810 15:19:46.895284 30335 leader_election.cc:302] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999dec29b8504c69a3296d798ca1086d; no voters:
I20260810 15:19:46.895371 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:46.895479 30335 leader_election.cc:288] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:46.895505 30338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:46.895589 30616 external_mini_cluster.cc:1428] Setting key 65d899d9d53f58c439c391c426feffda
I20260810 15:19:46.895677 30338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [term 1 LEADER]: Becoming Leader. State: Replica: 999dec29b8504c69a3296d798ca1086d, State: Running, Role: LEADER
I20260810 15:19:46.895737 30335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:46.895798 30338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } }
I20260810 15:19:46.896133 30338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999dec29b8504c69a3296d798ca1086d. Latest consensus state: current_term: 1 leader_uuid: "999dec29b8504c69a3296d798ca1086d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } } }
I20260810 15:19:46.896210 30338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:46.896368 30341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999dec29b8504c69a3296d798ca1086d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999dec29b8504c69a3296d798ca1086d" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33949 } } }
I20260810 15:19:46.896485 30341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:46.896875 30351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:46.897434 30351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:46.899545 30351 catalog_manager.cc:1347] Generated new cluster ID: dccab55ada3b4c9899dc3478d342c945
I20260810 15:19:46.899600 30351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:19:46 dist-test-slave-gjrz krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:46.908027 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:46.896294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.29.230.62:33949)
0810 15:19:46.896447 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:46.896458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:46.896585 (+ 127us) server_negotiation.cc:407] Connection header received
0810 15:19:46.896710 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:46.896717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:46.896772 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:46.897540 (+ 768us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:46.897707 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:46.899043 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:46.900084 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:46.900750 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:46.900863 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:46.903557 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:46.903561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:46.903561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:46.903574 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:46.906035 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:46.906470 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:46.906473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:46.906473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:46.906589 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:46.906788 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:46.906789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:46.906789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:46.907087 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:46.907242 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:46.907629 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 15:19:46.907791 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260810 15:19:46.908530 30351 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:46.909065 30351 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:46.912356 30351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: Generated new TSK 0
I20260810 15:19:46.912523 30351 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:19:46.941591 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:47.039170 30375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:47.039391 30375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:47.039417 30375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:47.040544 30375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:47.040588 30375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:47.040613 30375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:47.042093 30375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:47.042325 30375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:47.042536 30375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:47.047356 30387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.047367 30382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.047400 30375 server_base.cc:1034] running on GCE node
W20260810 15:19:47.047370 30380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.047765 30375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:47.047993 30375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:47.049167 30375 hybrid_clock.cc:648] HybridClock initialized: now 1786375187049151 us; error 31 us; skew 500 ppm
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:47.051802 30375 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:19:47.052233 30375 webserver.cc:460] Webserver started at http://127.29.230.1:43005/ using document root <none> and password file <none>
I20260810 15:19:47.052450 30375 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:47.052505 30375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:47.052630 30375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:47.053634 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "5c0f41a5afc1413897df8d3d53f7b748"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "a2ab3fcfdbe49455e629e806b978e0c1"
server_key_iv: "331819c0846fdd8d4feb6d9c288eef69"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.053949 30375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "5c0f41a5afc1413897df8d3d53f7b748"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "a2ab3fcfdbe49455e629e806b978e0c1"
server_key_iv: "331819c0846fdd8d4feb6d9c288eef69"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.055207 30375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:19:47.055929 30411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.056072 30375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:47.056156 30375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "5c0f41a5afc1413897df8d3d53f7b748"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "a2ab3fcfdbe49455e629e806b978e0c1"
server_key_iv: "331819c0846fdd8d4feb6d9c288eef69"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.056237 30375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:47.072186 30375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:47.074395 30375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:47.074543 30375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:47.074747 30375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:47.075038 30375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:47.075071 30375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.075098 30375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:47.075122 30375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.080598 30375 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:36579
I20260810 15:19:47.080642 30634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:36579 every 8 connection(s)
I20260810 15:19:47.080940 30375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:47.087354 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30375
I20260810 15:19:47.087457 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:47.087642 30616 external_mini_cluster.cc:1428] Setting key 888115e5f1cebe7fcc03c22c9352caeb
I20260810 15:19:47.089924 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.081631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:42305 (local address 127.29.230.62:33949)
0810 15:19:47.081696 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.081698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.081868 (+ 170us) server_negotiation.cc:407] Connection header received
0810 15:19:47.082378 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.082379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.082390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.082449 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:47.083363 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.083904 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.084910 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.085127 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.085185 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.087195 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:47.087196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:47.087196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:47.087205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:47.088665 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.089023 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.089024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.089025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.089085 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.089300 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.089301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.089301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.089416 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:47.089472 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.089737 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.089835 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 15:19:47.090184 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.081593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33949 (local address 127.29.230.1:42305)
0810 15:19:47.081738 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.081779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.082266 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.082520 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.082524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.083064 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:47.083255 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.083286 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.083987 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.083988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.084812 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.084814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.085201 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.085202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.085337 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.085786 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:47.085796 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:47.087099 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:47.088737 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.088745 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.088747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.088952 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.089143 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.089144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.089145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.089243 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.089471 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:47.089485 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:47.089616 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.089703 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.089857 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260810 15:19:47.090668 30636 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33949
I20260810 15:19:47.090758 30636 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:47.090934 30636 heartbeater.cc:511] Master 127.29.230.62:33949 requested a full tablet report, sending...
I20260810 15:19:47.091450 30271 ts_manager.cc:194] Registered new tserver with Master: 5c0f41a5afc1413897df8d3d53f7b748 (127.29.230.1:36579)
I20260810 15:19:47.092056 30271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:42305
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:19:47.113873 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:47.197386 30642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:47.197582 30642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:47.197604 30642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:47.198707 30642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:47.198747 30642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:47.198772 30642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:47.200244 30642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:47.200450 30642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:47.200650 30642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:47.205473 30650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.205437 30649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.205447 30652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.205502 30642 server_base.cc:1034] running on GCE node
I20260810 15:19:47.205780 30642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:47.206034 30642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:47.207187 30642 hybrid_clock.cc:648] HybridClock initialized: now 1786375187207177 us; error 32 us; skew 500 ppm
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:47.209651 30642 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:19:47.210083 30642 webserver.cc:460] Webserver started at http://127.29.230.2:35819/ using document root <none> and password file <none>
I20260810 15:19:47.210279 30642 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:47.210327 30642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:47.210448 30642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:47.211489 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "45339d8de6864109945fc8241df390c7"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "37c66265c9648e3186da22166325c0ca"
server_key_iv: "2c3478804597b2cbe385af3b484e08be"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.211805 30642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "45339d8de6864109945fc8241df390c7"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "37c66265c9648e3186da22166325c0ca"
server_key_iv: "2c3478804597b2cbe385af3b484e08be"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.213042 30642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:47.213724 30662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.213868 30642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:47.213949 30642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "45339d8de6864109945fc8241df390c7"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "37c66265c9648e3186da22166325c0ca"
server_key_iv: "2c3478804597b2cbe385af3b484e08be"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.214037 30642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:47.235471 30642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:47.237396 30642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:47.237531 30642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:47.237746 30642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:47.238083 30642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:47.238121 30642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.238148 30642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:47.238165 30642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.243559 30642 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:34215
I20260810 15:19:47.243624 30868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:34215 every 8 connection(s)
I20260810 15:19:47.243923 30642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:47.249402 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30642
I20260810 15:19:47.249516 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:47.249691 30616 external_mini_cluster.cc:1428] Setting key 1dec484fe34ea41bacf0083c490feae0
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:47.252861 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.244650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:33845 (local address 127.29.230.62:33949)
0810 15:19:47.244698 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.244700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.244871 (+ 171us) server_negotiation.cc:407] Connection header received
0810 15:19:47.245376 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.245378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.245389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.245441 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:47.246374 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.246882 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.247859 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.248100 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.248163 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.250186 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:47.250187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:47.250187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:47.250195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:47.251581 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.251939 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.251940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.251941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.252002 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.252232 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.252233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.252234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.252355 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:47.252415 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.252685 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.252774 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:19:47.253012 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.244606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33949 (local address 127.29.230.2:33845)
0810 15:19:47.244768 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.244805 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.245270 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.245517 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.245521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.246081 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:47.246275 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.246279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.246959 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.246960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.247783 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.247785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.248169 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.248170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.248267 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.248717 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:47.248726 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:47.250111 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:47.251656 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.251663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.251664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.251874 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.252065 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.252066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.252066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.252158 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.252411 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:47.252418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:47.252564 (+ 146us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.252655 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.252804 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260810 15:19:47.253439 30870 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33949
I20260810 15:19:47.253530 30870 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:47.253695 30870 heartbeater.cc:511] Master 127.29.230.62:33949 requested a full tablet report, sending...
I20260810 15:19:47.254053 30271 ts_manager.cc:194] Registered new tserver with Master: 45339d8de6864109945fc8241df390c7 (127.29.230.2:34215)
I20260810 15:19:47.254416 30271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:33845
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:19:47.274394 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:40625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:47.355382 30883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:47.355584 30883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:47.355607 30883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:47.356735 30883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:47.356778 30883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:47.356803 30883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:47.358223 30883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:40625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:47.358459 30883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:47.358673 30883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:47.363904 30894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.364022 30897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.363914 30895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.363967 30883 server_base.cc:1034] running on GCE node
I20260810 15:19:47.364328 30883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:47.364533 30883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:47.365680 30883 hybrid_clock.cc:648] HybridClock initialized: now 1786375187365668 us; error 30 us; skew 500 ppm
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:47.368165 30883 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:19:47.368619 30883 webserver.cc:460] Webserver started at http://127.29.230.3:38043/ using document root <none> and password file <none>
I20260810 15:19:47.368831 30883 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:47.368880 30883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:47.369002 30883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:47.369980 30883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "f957ca4d75494e9f8e9761323bd67f72"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "0f58f482fe6d87c0ac0bd786c82494c8"
server_key_iv: "d9733718d3bb95415faf3a29d8178699"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.370330 30883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "f957ca4d75494e9f8e9761323bd67f72"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "0f58f482fe6d87c0ac0bd786c82494c8"
server_key_iv: "d9733718d3bb95415faf3a29d8178699"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.371721 30883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:47.372506 30909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.372669 30883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:47.372754 30883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "f957ca4d75494e9f8e9761323bd67f72"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "0f58f482fe6d87c0ac0bd786c82494c8"
server_key_iv: "d9733718d3bb95415faf3a29d8178699"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.372822 30883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:47.386482 30883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:47.388238 30883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:47.388365 30883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:47.388581 30883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:47.388862 30883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:47.388895 30883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.388923 30883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:47.388947 30883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.394605 30883 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:36143
I20260810 15:19:47.394651 31106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:36143 every 8 connection(s)
I20260810 15:19:47.394934 30883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:47.399446 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 30883
I20260810 15:19:47.399554 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:47.399739 30616 external_mini_cluster.cc:1428] Setting key 2572dea8d447adea8621fdace20ebee2
Aug 10 15:19:47 dist-test-slave-gjrz krb5kdc[30191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375187, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:47.403844 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.395640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:43983 (local address 127.29.230.62:33949)
0810 15:19:47.395711 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.395713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.395871 (+ 158us) server_negotiation.cc:407] Connection header received
0810 15:19:47.396455 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.396456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.396470 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.396555 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:47.397551 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.398071 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.399022 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.399208 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.399257 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.401392 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:47.401393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:47.401394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:47.401400 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:47.402579 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.402961 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.402962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.402963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.403025 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.403313 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.403314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.403315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.403442 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:47.403500 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.403711 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.403776 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:19:47.403966 31112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.395599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33949 (local address 127.29.230.3:43983)
0810 15:19:47.395752 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.395785 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.396337 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.396608 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.396611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.397234 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:47.397473 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.397478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.398151 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.398151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.398915 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.398917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.399298 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.399299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.399421 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.399878 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:47.399887 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:47.401320 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:47.402649 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.402655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.402656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.402858 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.403097 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:47.403098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:47.403098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:47.403225 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:47.403494 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:47.403498 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:47.403607 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.403691 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.403806 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260810 15:19:47.404387 31107 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33949
I20260810 15:19:47.404465 31107 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:47.404623 31107 heartbeater.cc:511] Master 127.29.230.62:33949 requested a full tablet report, sending...
I20260810 15:19:47.404946 30271 ts_manager.cc:194] Registered new tserver with Master: f957ca4d75494e9f8e9761323bd67f72 (127.29.230.3:36143)
I20260810 15:19:47.405295 30271 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:43983
I20260810 15:19:47.411378 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:47.418145 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.412607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.29.230.62:33949)
0810 15:19:47.412654 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.412655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.412781 (+ 126us) server_negotiation.cc:407] Connection header received
0810 15:19:47.412821 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.412822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.412832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.412879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:47.413329 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.413788 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.414538 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.414717 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.414763 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.415992 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:47.415993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:47.415994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:47.416000 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:47.417101 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.417443 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.417443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.417444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.417492 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:47.417683 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:47.417684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:47.417684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:47.417810 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:47.417895 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.418037 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.418087 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 15:19:47.419497 30271 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:47.420980 30271 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:47.427567 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:36143 (local address 127.0.0.1:52620)
0810 15:19:47.423635 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.423674 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.423740 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.424196 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.424197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.424198 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:47.424237 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.424238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425536 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.425536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.426629 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.426631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427258 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.427362 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427376 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.427440 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.427492 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":39,"thread_start_us":70,"threads_started":1}
I20260810 15:19:47.427632 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.29.230.2:34215)
0810 15:19:47.423800 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.423804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.423993 (+ 189us) server_negotiation.cc:407] Connection header received
0810 15:19:47.424036 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.424039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.424053 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.424127 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:47.424313 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425468 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.426545 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427329 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427396 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427452 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.427537 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.427570 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260810 15:19:47.427704 31130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:34215 (local address 127.0.0.1:51808)
0810 15:19:47.423933 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.423940 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.423972 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.424157 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.424161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.424163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:47.424227 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.424229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425583 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.425584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.426450 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.426453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427420 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.427476 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427479 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.427526 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.427558 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":96,"threads_started":1}
I20260810 15:19:47.427899 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.29.230.1:36579)
0810 15:19:47.423644 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.423648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.423882 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:19:47.423932 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.423934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.423948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.424042 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:47.424516 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425193 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.426627 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427329 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427399 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427452 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.427757 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.427813 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260810 15:19:47.428050 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.29.230.3:36143)
0810 15:19:47.424051 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.424054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.424062 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:19:47.424092 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.424094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.424106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.424306 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:47.424373 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425421 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.426688 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427177 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427240 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427332 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.427942 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.427987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":72,"threads_started":1}
I20260810 15:19:47.428359 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.423363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:36579 (local address 127.0.0.1:38354)
0810 15:19:47.423830 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.423838 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.423882 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.424281 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.424282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.424284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:47.424384 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.424386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.425283 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.425284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.426550 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.426553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.427600 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.427601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.427654 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.427657 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.428257 (+ 600us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.428295 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":33,"spinlock_wait_cycles":384,"thread_start_us":95,"threads_started":1}
I20260810 15:19:47.428467 30764 tablet_service.cc:1467] Processing CreateTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 (DEFAULT_TABLE table=test-table [id=69c985dd92c74aa1b0b385cad75268bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:47.428735 30995 tablet_service.cc:1467] Processing CreateTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 (DEFAULT_TABLE table=test-table [id=69c985dd92c74aa1b0b385cad75268bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:47.428864 30764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ebc413fe6410a9093ccea9c1a3795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:47.429045 30995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ebc413fe6410a9093ccea9c1a3795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:47.431324 31136 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Bootstrap starting.
I20260810 15:19:47.431468 31137 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Bootstrap starting.
I20260810 15:19:47.431773 30490 tablet_service.cc:1467] Processing CreateTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 (DEFAULT_TABLE table=test-table [id=69c985dd92c74aa1b0b385cad75268bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:47.432077 31137 tablet_bootstrap.cc:654] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:47.432094 30490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 290ebc413fe6410a9093ccea9c1a3795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:47.432091 31136 tablet_bootstrap.cc:654] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:47.432554 31137 log.cc:824] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:47.432601 31136 log.cc:824] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:47.433209 31137 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: No bootstrap required, opened a new log
I20260810 15:19:47.433272 31137 ts_tablet_manager.cc:1397] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:47.434572 31142 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Bootstrap starting.
I20260810 15:19:47.434538 31137 raft_consensus.cc:348] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.434710 31137 raft_consensus.cc:374] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:47.434749 31137 raft_consensus.cc:729] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f957ca4d75494e9f8e9761323bd67f72, State: Initialized, Role: FOLLOWER
I20260810 15:19:47.434855 31137 consensus_queue.cc:260] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.435127 31137 ts_tablet_manager.cc:1428] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:19:47.435175 31142 tablet_bootstrap.cc:654] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:47.435129 31107 heartbeater.cc:503] Master 127.29.230.62:33949 was elected leader, sending a full tablet report...
I20260810 15:19:47.435565 31136 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: No bootstrap required, opened a new log
I20260810 15:19:47.435628 31136 ts_tablet_manager.cc:1397] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 15:19:47.435745 31142 log.cc:824] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:47.436472 31142 tablet_bootstrap.cc:492] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: No bootstrap required, opened a new log
I20260810 15:19:47.436533 31142 ts_tablet_manager.cc:1397] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:19:47.437219 31136 raft_consensus.cc:348] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.437351 31136 raft_consensus.cc:374] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:47.437382 31136 raft_consensus.cc:729] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45339d8de6864109945fc8241df390c7, State: Initialized, Role: FOLLOWER
I20260810 15:19:47.437491 31136 consensus_queue.cc:260] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.437722 31136 ts_tablet_manager.cc:1428] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 15:19:47.437762 30870 heartbeater.cc:503] Master 127.29.230.62:33949 was elected leader, sending a full tablet report...
I20260810 15:19:47.437893 31142 raft_consensus.cc:348] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.438162 31142 raft_consensus.cc:374] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:47.438213 31142 raft_consensus.cc:729] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0f41a5afc1413897df8d3d53f7b748, State: Initialized, Role: FOLLOWER
I20260810 15:19:47.438316 31142 consensus_queue.cc:260] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.438577 31142 ts_tablet_manager.cc:1428] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 15:19:47.438591 30636 heartbeater.cc:503] Master 127.29.230.62:33949 was elected leader, sending a full tablet report...
W20260810 15:19:47.494458 30871 tablet.cc:2369] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:19:47.581674 30638 tablet.cc:2369] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 15:19:47.645535 31108 tablet.cc:2369] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:47.763070 31143 raft_consensus.cc:482] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:47.763204 31143 raft_consensus.cc:504] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.763633 31143 leader_election.cc:288] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c0f41a5afc1413897df8d3d53f7b748 (127.29.230.1:36579), 45339d8de6864109945fc8241df390c7 (127.29.230.2:34215)
I20260810 15:19:47.766366 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.763939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:41361 (local address 127.29.230.1:36579)
0810 15:19:47.764016 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.764018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.764081 (+ 63us) server_negotiation.cc:407] Connection header received
0810 15:19:47.764140 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.764141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.764151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.764210 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:47.764352 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.764804 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.765702 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.766045 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.766110 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.766119 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.766256 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.766294 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 15:19:47.766371 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.763898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:54681 (local address 127.29.230.2:34215)
0810 15:19:47.763943 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.763944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.763956 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:19:47.764054 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.764056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.764067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.764125 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:47.764286 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.764783 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.765707 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.766061 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.766123 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.766131 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.766278 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.766311 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 15:19:47.766378 31112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.763831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:34215 (local address 127.29.230.3:54681)
0810 15:19:47.763910 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.763922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.763968 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.764135 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.764136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.764138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:47.764215 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.764217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.764856 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.764857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.765612 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.765615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.766131 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.766132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.766211 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.766213 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.766263 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.766301 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260810 15:19:47.766377 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.763847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:36579 (local address 127.29.230.3:41361)
0810 15:19:47.764010 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.764021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:47.764087 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:47.764218 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:47.764219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:47.764220 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:47.764293 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.764294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.764884 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.764885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.765617 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:47.765619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.766127 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.766128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:47.766196 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.766199 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:47.766243 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:19:47.766284 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260810 15:19:47.766757 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ebc413fe6410a9093ccea9c1a3795" candidate_uuid: "f957ca4d75494e9f8e9761323bd67f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45339d8de6864109945fc8241df390c7" is_pre_election: true
I20260810 15:19:47.766922 30796 raft_consensus.cc:2454] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f957ca4d75494e9f8e9761323bd67f72 in term 0.
I20260810 15:19:47.766901 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ebc413fe6410a9093ccea9c1a3795" candidate_uuid: "f957ca4d75494e9f8e9761323bd67f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c0f41a5afc1413897df8d3d53f7b748" is_pre_election: true
I20260810 15:19:47.767024 30536 raft_consensus.cc:2454] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f957ca4d75494e9f8e9761323bd67f72 in term 0.
I20260810 15:19:47.767172 30911 leader_election.cc:302] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45339d8de6864109945fc8241df390c7, f957ca4d75494e9f8e9761323bd67f72; no voters:
I20260810 15:19:47.767331 31143 raft_consensus.cc:2781] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:47.767393 31143 raft_consensus.cc:482] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:47.767413 31143 raft_consensus.cc:3037] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:47.768114 31143 raft_consensus.cc:504] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.768229 31143 leader_election.cc:288] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [CANDIDATE]: Term 1 election: Requested vote from peers 5c0f41a5afc1413897df8d3d53f7b748 (127.29.230.1:36579), 45339d8de6864109945fc8241df390c7 (127.29.230.2:34215)
I20260810 15:19:47.768392 30536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ebc413fe6410a9093ccea9c1a3795" candidate_uuid: "f957ca4d75494e9f8e9761323bd67f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c0f41a5afc1413897df8d3d53f7b748"
I20260810 15:19:47.768414 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "290ebc413fe6410a9093ccea9c1a3795" candidate_uuid: "f957ca4d75494e9f8e9761323bd67f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45339d8de6864109945fc8241df390c7"
I20260810 15:19:47.768476 30796 raft_consensus.cc:3037] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:47.768468 30536 raft_consensus.cc:3037] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:47.769172 30796 raft_consensus.cc:2454] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f957ca4d75494e9f8e9761323bd67f72 in term 1.
I20260810 15:19:47.769208 30536 raft_consensus.cc:2454] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f957ca4d75494e9f8e9761323bd67f72 in term 1.
I20260810 15:19:47.769325 30911 leader_election.cc:302] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45339d8de6864109945fc8241df390c7, f957ca4d75494e9f8e9761323bd67f72; no voters:
I20260810 15:19:47.769459 31143 raft_consensus.cc:2781] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:47.769593 31143 raft_consensus.cc:686] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 1 LEADER]: Becoming Leader. State: Replica: f957ca4d75494e9f8e9761323bd67f72, State: Running, Role: LEADER
I20260810 15:19:47.769670 31143 consensus_queue.cc:237] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } }
I20260810 15:19:47.770447 30271 catalog_manager.cc:5495] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to f957ca4d75494e9f8e9761323bd67f72 (127.29.230.3). New cstate: current_term: 1 leader_uuid: "f957ca4d75494e9f8e9761323bd67f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f957ca4d75494e9f8e9761323bd67f72" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 36143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 } health_report { overall_health: UNKNOWN } } }
I20260810 15:19:47.782277 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.779982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.29.230.3:36143)
0810 15:19:47.780021 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.780023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.780079 (+ 56us) server_negotiation.cc:407] Connection header received
0810 15:19:47.780147 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.780148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.780166 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.780230 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:47.780347 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.780810 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.781549 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.781747 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.781800 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.781971 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:47.782067 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:47.782152 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.782196 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.782206 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:19:47.783892 30796 raft_consensus.cc:1261] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 1 FOLLOWER]: Refusing update from remote peer f957ca4d75494e9f8e9761323bd67f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:47.783892 30536 raft_consensus.cc:1261] T 290ebc413fe6410a9093ccea9c1a3795 P 5c0f41a5afc1413897df8d3d53f7b748 [term 1 FOLLOWER]: Refusing update from remote peer f957ca4d75494e9f8e9761323bd67f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:47.784204 31152 consensus_queue.cc:1035] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c0f41a5afc1413897df8d3d53f7b748" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 36579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:47.784319 31143 consensus_queue.cc:1035] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45339d8de6864109945fc8241df390c7" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 34215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:47.788321 31158 mvcc.cc:205] Tried to move back new op lower bound from 7316992769160601600 to 7316992769104998400. Current Snapshot: MvccSnapshot[applied={T|T < 7316992769160601600 or (T in {7316992769160601600})}]
I20260810 15:19:47.788620 30270 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:47.788702 30270 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:47.789340 30270 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999dec29b8504c69a3296d798ca1086d: Sending DeleteTablet for 3 replicas of tablet 290ebc413fe6410a9093ccea9c1a3795
I20260810 15:19:47.789695 30764 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:47 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:51808
I20260810 15:19:47.789695 30995 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:47 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:52620
I20260810 15:19:47.789824 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30375
I20260810 15:19:47.789906 30490 tablet_service.cc:1514] Processing DeleteTablet for tablet 290ebc413fe6410a9093ccea9c1a3795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:47 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:38354
W20260810 15:19:47.807812 30252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:36579 (error 108)
I20260810 15:19:47.807919 31169 tablet_replica.cc:330] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7: stopping tablet replica
I20260810 15:19:47.808048 31169 raft_consensus.cc:2229] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:47.808090 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30642
I20260810 15:19:47.808190 31169 raft_consensus.cc:2258] T 290ebc413fe6410a9093ccea9c1a3795 P 45339d8de6864109945fc8241df390c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:47.808362 31168 tablet_replica.cc:330] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: stopping tablet replica
I20260810 15:19:47.808459 31168 raft_consensus.cc:2229] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:47.808552 31168 raft_consensus.cc:2258] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:47.809051 31168 ts_tablet_manager.cc:1905] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:47.810774 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.808400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:36579 (local address 127.0.0.1:38358)
0810 15:19:47.810683 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.810737 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:36579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2254}
W20260810 15:19:47.811499 30252 catalog_manager.cc:4584] TS 5c0f41a5afc1413897df8d3d53f7b748 (127.29.230.1:36579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 290ebc413fe6410a9093ccea9c1a3795: Network error: Client connection negotiation failed: client connection to 127.29.230.1:36579: connect: Connection refused (error 111)
I20260810 15:19:47.812038 31168 ts_tablet_manager.cc:1918] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:47.812114 31168 log.cc:1196] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/290ebc413fe6410a9093ccea9c1a3795
I20260810 15:19:47.812369 31168 ts_tablet_manager.cc:1939] T 290ebc413fe6410a9093ccea9c1a3795 P f957ca4d75494e9f8e9761323bd67f72: Deleting consensus metadata
I20260810 15:19:47.812824 30251 catalog_manager.cc:4841] TS f957ca4d75494e9f8e9761323bd67f72 (127.29.230.3:36143): tablet 290ebc413fe6410a9093ccea9c1a3795 (table test-table [id=69c985dd92c74aa1b0b385cad75268bd]) successfully deleted
I20260810 15:19:47.815521 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30883
I20260810 15:19:47.815585 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.815482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:34215 (local address 127.0.0.1:51824)
0810 15:19:47.815535 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:47.815558 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.2:34215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:19:47.830570 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 30217
2026-08-10T15:19:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2373 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T15:19:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:47Z Disabled control of system clock
I20260810 15:19:47.874490 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33387
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:47.955614 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:47.955840 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:47.955871 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:47.957058 31176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:47.957101 31176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:47.957118 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:47.957134 31176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:47.957149 31176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:47.958479 31176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:33387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:47.958715 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:47.958989 31176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:47.963768 31189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:47.963773 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.963814 31176 server_base.cc:1034] running on GCE node
W20260810 15:19:47.963768 31184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:47.964203 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:47.964475 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:47.965632 31176 hybrid_clock.cc:648] HybridClock initialized: now 1786375187965618 us; error 30 us; skew 500 ppm
I20260810 15:19:47.970130 31176 webserver.cc:460] Webserver started at http://127.29.230.62:44441/ using document root <none> and password file <none>
I20260810 15:19:47.970386 31176 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:47.970435 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:47.970572 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:47.971724 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "e1c1eb778a574bcd9f2d64bc54dc6108"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "2fa6cc8f3101a33434ce907e608898c0"
server_key_iv: "c1000a97b5fd8617277ebbfa07cee9d2"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.972084 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "e1c1eb778a574bcd9f2d64bc54dc6108"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "2fa6cc8f3101a33434ce907e608898c0"
server_key_iv: "c1000a97b5fd8617277ebbfa07cee9d2"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.973423 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:47.974205 31199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:47.974359 31176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:47.974449 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e1c1eb778a574bcd9f2d64bc54dc6108"
format_stamp: "Formatted at 2026-08-10 15:19:47 on dist-test-slave-gjrz"
server_key: "2fa6cc8f3101a33434ce907e608898c0"
server_key_iv: "c1000a97b5fd8617277ebbfa07cee9d2"
server_key_version: "encryptionkey@0"
I20260810 15:19:47.974524 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:47.982384 31176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:47.982641 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:47.982764 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:47.986372 31176 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:33387
I20260810 15:19:47.986419 31288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:33387 every 8 connection(s)
I20260810 15:19:47.986701 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:47.987695 31290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:47.988987 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31176
I20260810 15:19:47.989090 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:47.989262 30616 external_mini_cluster.cc:1428] Setting key 058ce6a51b2b891e1ee4ba544aa2b2ea
I20260810 15:19:47.990180 31290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: Bootstrap starting.
I20260810 15:19:47.990818 31290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:47.991104 31290 log.cc:824] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:47.991762 31290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: No bootstrap required, opened a new log
I20260810 15:19:47.993966 31290 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } }
I20260810 15:19:47.994143 31290 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:47.994190 31290 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1c1eb778a574bcd9f2d64bc54dc6108, State: Initialized, Role: FOLLOWER
I20260810 15:19:47.994329 31290 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } }
I20260810 15:19:47.994417 31290 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:47.994451 31290 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:47.994499 31290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:47.994939 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:47.989744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.29.230.62:33387)
0810 15:19:47.989912 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:47.989918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:47.989942 (+ 24us) server_negotiation.cc:407] Connection header received
0810 15:19:47.990363 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:47.990370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:47.990430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:47.990558 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:47.990788 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.991759 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.992842 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:47.993286 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:47.993381 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:47.994055 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:47.994058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:47.994059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:47.994322 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:47.994366 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:47.994381 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:47.994629 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:47.994662 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 15:19:47.994744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260810 15:19:47.995919 31290 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } }
I20260810 15:19:47.996030 31290 leader_election.cc:302] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1c1eb778a574bcd9f2d64bc54dc6108; no voters:
I20260810 15:19:47.996189 31290 leader_election.cc:288] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:47.996285 31297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:47.996426 31297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [term 1 LEADER]: Becoming Leader. State: Replica: e1c1eb778a574bcd9f2d64bc54dc6108, State: Running, Role: LEADER
I20260810 15:19:47.996439 31290 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:47.996555 31297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } }
I20260810 15:19:47.997018 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } } }
I20260810 15:19:47.997152 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:47.997018 31297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1c1eb778a574bcd9f2d64bc54dc6108. Latest consensus state: current_term: 1 leader_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1c1eb778a574bcd9f2d64bc54dc6108" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33387 } } }
I20260810 15:19:47.997399 31297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:47.997416 31313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:47.997890 31313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:47.999610 31313 catalog_manager.cc:1347] Generated new cluster ID: a3e414d89ecc40f59688956bf9d439b1
I20260810 15:19:47.999655 31313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:48.023069 31313 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:48.023654 31313 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:48.030309 31313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: Generated new TSK 0
I20260810 15:19:48.030522 31313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:48.034178 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--builtin_ntp_servers=127.29.230.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:48.127820 31325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:48.128010 31325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:48.128031 31325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:48.129148 31325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:48.129189 31325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:48.129212 31325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:48.130594 31325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:48.130808 31325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:48.131089 31325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:48.136349 31339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:48.136379 31334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:48.136356 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:48.136632 31325 server_base.cc:1034] running on GCE node
I20260810 15:19:48.136842 31325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:48.137066 31325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:48.138217 31325 hybrid_clock.cc:648] HybridClock initialized: now 1786375188138203 us; error 31 us; skew 500 ppm
I20260810 15:19:48.139508 31325 webserver.cc:460] Webserver started at http://127.29.230.1:39265/ using document root <none> and password file <none>
I20260810 15:19:48.139726 31325 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:48.139775 31325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:48.139923 31325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:48.141055 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "964112d582014c439911bbfe000fd105"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ae3ba72bb8a563099d7c6a2bec001610"
server_key_iv: "19b2900387045163d5693ebfb820e71d"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.141394 31325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "964112d582014c439911bbfe000fd105"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ae3ba72bb8a563099d7c6a2bec001610"
server_key_iv: "19b2900387045163d5693ebfb820e71d"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.142659 31325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:48.143383 31352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.143520 31325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:48.143610 31325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "964112d582014c439911bbfe000fd105"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ae3ba72bb8a563099d7c6a2bec001610"
server_key_iv: "19b2900387045163d5693ebfb820e71d"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.143671 31325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:48.154438 31325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:48.154721 31325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:48.154838 31325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:48.155040 31325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:48.155375 31325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:48.155412 31325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.155439 31325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:48.155463 31325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.160821 31325 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:41457
I20260810 15:19:48.160887 31549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:41457 every 8 connection(s)
I20260810 15:19:48.161175 31325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:48.166074 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.161856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:34241 (local address 127.29.230.62:33387)
0810 15:19:48.161920 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.161923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.162151 (+ 228us) server_negotiation.cc:407] Connection header received
0810 15:19:48.162655 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.162656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.162670 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.162723 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:48.163396 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.163897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.164854 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.165044 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.165094 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.165872 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:48.165873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:48.165873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:48.165877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:48.165891 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:48.165895 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:48.165934 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.165990 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.166014 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:19:48.166266 31555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.161838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33387 (local address 127.29.230.1:34241)
0810 15:19:48.162028 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.162067 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.162575 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.162776 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.162779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.162787 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:48.163318 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.163325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.163959 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.163960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.164776 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.164779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.165149 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.165150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.165276 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.165740 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:48.165741 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:48.165773 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:48.165797 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:48.165940 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:48.165943 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:48.165945 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.165991 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.166075 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260810 15:19:48.166808 31552 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33387
I20260810 15:19:48.166893 31552 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:48.167053 31552 heartbeater.cc:511] Master 127.29.230.62:33387 requested a full tablet report, sending...
I20260810 15:19:48.167547 31231 ts_manager.cc:194] Registered new tserver with Master: 964112d582014c439911bbfe000fd105 (127.29.230.1:41457)
I20260810 15:19:48.168121 31231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.1:34241
I20260810 15:19:48.170532 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31325
I20260810 15:19:48.170619 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:48.170773 30616 external_mini_cluster.cc:1428] Setting key 84118d01928f4923b7564001c62a3c3a
I20260810 15:19:48.171633 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--builtin_ntp_servers=127.29.230.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:48.252633 31556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:48.252831 31556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:48.252851 31556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:48.253911 31556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:48.253942 31556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:48.253964 31556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:48.255234 31556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:48.255434 31556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:48.255666 31556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:48.260411 31570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:48.260448 31567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:48.260501 31556 server_base.cc:1034] running on GCE node
W20260810 15:19:48.260422 31562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:48.260766 31556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:48.261001 31556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:48.262178 31556 hybrid_clock.cc:648] HybridClock initialized: now 1786375188262153 us; error 33 us; skew 500 ppm
I20260810 15:19:48.263402 31556 webserver.cc:460] Webserver started at http://127.29.230.2:35355/ using document root <none> and password file <none>
I20260810 15:19:48.263629 31556 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:48.263677 31556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:48.263813 31556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:48.264814 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "3f30ee1a2f3940be8f35612c1d9e48b4"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ed1ea691df52a6171cd6dac48793dd99"
server_key_iv: "512b4878e7b04d9977c27c6537664a86"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.265194 31556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "3f30ee1a2f3940be8f35612c1d9e48b4"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ed1ea691df52a6171cd6dac48793dd99"
server_key_iv: "512b4878e7b04d9977c27c6537664a86"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.266438 31556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:19:48.267100 31585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.267251 31556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:19:48.267365 31556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "3f30ee1a2f3940be8f35612c1d9e48b4"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "ed1ea691df52a6171cd6dac48793dd99"
server_key_iv: "512b4878e7b04d9977c27c6537664a86"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.267433 31556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:48.294124 31556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:48.294430 31556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:48.294557 31556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:48.294788 31556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:48.295122 31556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:48.295156 31556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.295185 31556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:48.295202 31556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.300890 31556 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:38829
I20260810 15:19:48.300930 31784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:38829 every 8 connection(s)
I20260810 15:19:48.301242 31556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:48.306210 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.301963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:57057 (local address 127.29.230.62:33387)
0810 15:19:48.302017 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.302019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.302261 (+ 242us) server_negotiation.cc:407] Connection header received
0810 15:19:48.302723 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.302724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.302734 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.302783 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:48.303433 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.303958 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.304987 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.305186 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.305241 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.306006 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:48.306007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:48.306008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:48.306012 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:48.306026 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:48.306029 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:48.306072 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.306131 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.306152 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:19:48.306402 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.301945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33387 (local address 127.29.230.2:57057)
0810 15:19:48.302145 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.302180 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.302648 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.302837 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.302840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.302848 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:48.303357 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.303362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.304030 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.304030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.304881 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.304884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.305260 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.305261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.305376 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.305878 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:48.305878 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:48.305908 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:48.305931 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:48.306078 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:48.306080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:48.306083 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.306131 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.306209 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260810 15:19:48.306851 31786 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33387
I20260810 15:19:48.306931 31786 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:48.307143 31786 heartbeater.cc:511] Master 127.29.230.62:33387 requested a full tablet report, sending...
I20260810 15:19:48.307348 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31556
I20260810 15:19:48.307451 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:48.307624 30616 external_mini_cluster.cc:1428] Setting key c7348cbbf5788c3d36fcf0eeadb9f7b3
I20260810 15:19:48.308028 31227 ts_manager.cc:194] Registered new tserver with Master: 3f30ee1a2f3940be8f35612c1d9e48b4 (127.29.230.2:38829)
I20260810 15:19:48.308449 31227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.2:57057
I20260810 15:19:48.308529 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--builtin_ntp_servers=127.29.230.20:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:48.390048 31789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:48.390237 31789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:48.390256 31789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:48.391351 31789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:48.391387 31789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:48.391410 31789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:48.392691 31789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:48.392869 31789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:48.393106 31789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:19:48.398130 31789 server_base.cc:1034] running on GCE node
W20260810 15:19:48.398132 31797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:48.398113 31796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:48.398113 31806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:48.398463 31789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:48.398685 31789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:48.399827 31789 hybrid_clock.cc:648] HybridClock initialized: now 1786375188399814 us; error 30 us; skew 500 ppm
I20260810 15:19:48.401115 31789 webserver.cc:460] Webserver started at http://127.29.230.3:42779/ using document root <none> and password file <none>
I20260810 15:19:48.401322 31789 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:48.401371 31789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:48.401494 31789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:48.402500 31789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "89af51f328514c008db5fb2f58b6c233"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "c6eba1fd20d2c74faee74f11d99e9b14"
server_key_iv: "b6737ed458cb5d41d128d1bf4e588577"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.402822 31789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "89af51f328514c008db5fb2f58b6c233"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "c6eba1fd20d2c74faee74f11d99e9b14"
server_key_iv: "b6737ed458cb5d41d128d1bf4e588577"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.404184 31789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:19:48.404896 31814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.405066 31789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:19:48.405150 31789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "89af51f328514c008db5fb2f58b6c233"
format_stamp: "Formatted at 2026-08-10 15:19:48 on dist-test-slave-gjrz"
server_key: "c6eba1fd20d2c74faee74f11d99e9b14"
server_key_iv: "b6737ed458cb5d41d128d1bf4e588577"
server_key_version: "encryptionkey@0"
I20260810 15:19:48.405212 31789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:48.429144 31789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:48.429455 31789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:48.429590 31789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:48.429816 31789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:48.430135 31789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:48.430171 31789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.430198 31789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:48.430217 31789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:48.435745 31789 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:44375
I20260810 15:19:48.435825 32016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:44375 every 8 connection(s)
I20260810 15:19:48.436113 31789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:48.441217 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.436848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:57443 (local address 127.29.230.62:33387)
0810 15:19:48.436941 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.436943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.437116 (+ 173us) server_negotiation.cc:407] Connection header received
0810 15:19:48.437646 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.437648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.437660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.437709 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:48.438385 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.438883 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.439918 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.440108 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.440166 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.440958 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:48.440960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:48.440960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:48.440965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:48.440980 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:48.440984 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:48.441026 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.441110 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.441139 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:19:48.441427 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.436788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33387 (local address 127.29.230.3:57443)
0810 15:19:48.437001 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.437035 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.437549 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.437783 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.437787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.437795 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:48.438295 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.438300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.438983 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.438983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.439825 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.439827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.440182 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.440183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.440313 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.440816 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:48.440817 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:48.440846 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:48.440869 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:48.441033 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:48.441038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:48.441041 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.441104 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.441208 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260810 15:19:48.441884 32017 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33387
I20260810 15:19:48.441979 32017 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:48.442154 32017 heartbeater.cc:511] Master 127.29.230.62:33387 requested a full tablet report, sending...
I20260810 15:19:48.442543 31227 ts_manager.cc:194] Registered new tserver with Master: 89af51f328514c008db5fb2f58b6c233 (127.29.230.3:44375)
I20260810 15:19:48.442891 31227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.3:57443
I20260810 15:19:48.444129 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 31789
I20260810 15:19:48.444222 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:48.444392 30616 external_mini_cluster.cc:1428] Setting key ecc18bd70af8ed6584cd653bf3b4b13e
I20260810 15:19:48.445381 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:48.449618 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.446741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.29.230.62:33387)
0810 15:19:48.446779 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.446781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.446891 (+ 110us) server_negotiation.cc:407] Connection header received
0810 15:19:48.446953 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.446954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.446963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.447037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:48.447201 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.447708 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.448447 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.448638 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.448691 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.449352 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:48.449354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:48.449354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:48.449358 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:48.449372 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:48.449376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:48.449453 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.449529 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.449539 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:19:48.451229 31227 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:48.452780 31227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:48.459064 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.455053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.29.230.1:41457)
0810 15:19:48.455247 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.455251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.455286 (+ 35us) server_negotiation.cc:407] Connection header received
0810 15:19:48.455395 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.455398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.455416 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.455491 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:48.455666 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.456627 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.457913 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.458645 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458724 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.458821 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.458932 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.458975 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260810 15:19:48.459074 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.454966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:41457 (local address 127.0.0.1:50052)
0810 15:19:48.455169 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.455211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.455304 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.455504 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.455507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.455509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:48.455583 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.455584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.456720 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.456721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.457827 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.457830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.458734 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.458816 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.458830 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.458915 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.458972 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260810 15:19:48.459236 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.455370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.29.230.2:38829)
0810 15:19:48.455530 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.455534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.455544 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:48.455582 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.455584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.455599 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.455676 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:48.455944 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.456720 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458006 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.458704 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458760 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.458830 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.459116 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.459163 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260810 15:19:48.459486 32028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.455186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:44375 (local address 127.0.0.1:45440)
0810 15:19:48.455808 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.455817 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.455888 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.456117 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.456118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.456119 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:48.456155 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.456156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.457331 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.457332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.458253 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.458255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.459191 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.459192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.459255 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.459259 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.459385 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.459425 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":70,"threads_started":1}
I20260810 15:19:48.459515 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.455569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.29.230.3:44375)
0810 15:19:48.455753 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.455757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.455865 (+ 108us) server_negotiation.cc:407] Connection header received
0810 15:19:48.455970 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.455973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.455987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.456036 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:48.456231 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.457246 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458323 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.458884 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458942 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.459023 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.459402 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.459443 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260810 15:19:48.459908 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.454966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:38829 (local address 127.0.0.1:51780)
0810 15:19:48.455328 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.455337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.455384 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.455797 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.455798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.455800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:48.455847 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.455848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.456795 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.456796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.457918 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.457921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.458865 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.458866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.458931 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.458935 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.458981 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.459018 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260810 15:19:48.459868 31438 tablet_service.cc:1467] Processing CreateTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db (DEFAULT_TABLE table=test-table [id=831a1a1a5adf433f925ac89ab4e1ced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:48.460282 31438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f643a3ef2e5f45a38e675b58b52ee7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:48.460204 31898 tablet_service.cc:1467] Processing CreateTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db (DEFAULT_TABLE table=test-table [id=831a1a1a5adf433f925ac89ab4e1ced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:48.460542 31898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f643a3ef2e5f45a38e675b58b52ee7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:48.460534 31658 tablet_service.cc:1467] Processing CreateTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db (DEFAULT_TABLE table=test-table [id=831a1a1a5adf433f925ac89ab4e1ced8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:48.460868 31658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f643a3ef2e5f45a38e675b58b52ee7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:48.463035 32037 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Bootstrap starting.
I20260810 15:19:48.463774 32037 tablet_bootstrap.cc:654] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:48.463910 32039 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: Bootstrap starting.
I20260810 15:19:48.464110 32037 log.cc:824] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:48.464113 32042 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Bootstrap starting.
I20260810 15:19:48.464645 32039 tablet_bootstrap.cc:654] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:48.464708 32042 tablet_bootstrap.cc:654] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:48.464937 32037 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: No bootstrap required, opened a new log
I20260810 15:19:48.464942 32039 log.cc:824] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:48.464967 32042 log.cc:824] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:48.465010 32037 ts_tablet_manager.cc:1397] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:48.465687 32039 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: No bootstrap required, opened a new log
I20260810 15:19:48.465764 32039 ts_tablet_manager.cc:1397] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:48.467559 32037 raft_consensus.cc:348] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.467732 32037 raft_consensus.cc:374] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:48.467746 32042 tablet_bootstrap.cc:492] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: No bootstrap required, opened a new log
I20260810 15:19:48.467765 32037 raft_consensus.cc:729] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89af51f328514c008db5fb2f58b6c233, State: Initialized, Role: FOLLOWER
I20260810 15:19:48.467813 32042 ts_tablet_manager.cc:1397] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 15:19:48.467885 32037 consensus_queue.cc:260] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.468166 32037 ts_tablet_manager.cc:1428] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 15:19:48.468189 32017 heartbeater.cc:503] Master 127.29.230.62:33387 was elected leader, sending a full tablet report...
I20260810 15:19:48.468197 32039 raft_consensus.cc:348] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.468333 32039 raft_consensus.cc:374] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:48.468353 32039 raft_consensus.cc:729] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964112d582014c439911bbfe000fd105, State: Initialized, Role: FOLLOWER
I20260810 15:19:48.468437 32039 consensus_queue.cc:260] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.468686 32039 ts_tablet_manager.cc:1428] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:48.468798 31552 heartbeater.cc:503] Master 127.29.230.62:33387 was elected leader, sending a full tablet report...
I20260810 15:19:48.470184 32042 raft_consensus.cc:348] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.470326 32042 raft_consensus.cc:374] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:48.470358 32042 raft_consensus.cc:729] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f30ee1a2f3940be8f35612c1d9e48b4, State: Initialized, Role: FOLLOWER
I20260810 15:19:48.470435 32042 consensus_queue.cc:260] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.470615 32042 ts_tablet_manager.cc:1428] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:48.470685 31786 heartbeater.cc:503] Master 127.29.230.62:33387 was elected leader, sending a full tablet report...
W20260810 15:19:48.551762 31787 tablet.cc:2369] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:48.570397 32050 raft_consensus.cc:482] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:48.570559 32050 raft_consensus.cc:504] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.570978 32050 leader_election.cc:288] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f30ee1a2f3940be8f35612c1d9e48b4 (127.29.230.2:38829), 964112d582014c439911bbfe000fd105 (127.29.230.1:41457)
I20260810 15:19:48.573973 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.571313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:56591 (local address 127.29.230.2:38829)
0810 15:19:48.571365 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.571367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.571375 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:19:48.571403 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.571404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.571429 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.571476 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:48.571623 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.572122 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573096 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.573645 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573719 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.573727 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.573868 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.573906 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:19:48.573988 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.571291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:36103 (local address 127.29.230.1:41457)
0810 15:19:48.571374 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.571377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.571504 (+ 127us) server_negotiation.cc:407] Connection header received
0810 15:19:48.571558 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.571559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.571570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.571616 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:48.571759 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.572247 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573149 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.573676 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573739 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.573746 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.573886 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.573933 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:19:48.573997 32019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.571157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:38829 (local address 127.29.230.3:56591)
0810 15:19:48.571214 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.571227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.571325 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.571486 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.571488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.571489 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:48.571560 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.571562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.572198 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.572199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.573006 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.573008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.573718 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.573793 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.573797 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.573855 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.573897 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":32}
I20260810 15:19:48.574025 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.571157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:41457 (local address 127.29.230.3:36103)
0810 15:19:48.571431 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.571440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:48.571501 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:48.571622 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:48.571622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:48.571624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:48.571692 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.571694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.572318 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.572319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.573074 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:48.573077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.573745 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.573746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:48.573819 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.573821 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:19:48.573866 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:19:48.573897 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":79,"thread_start_us":120,"threads_started":1}
I20260810 15:19:48.574333 31695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f643a3ef2e5f45a38e675b58b52ee7db" candidate_uuid: "89af51f328514c008db5fb2f58b6c233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" is_pre_election: true
I20260810 15:19:48.574363 31470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f643a3ef2e5f45a38e675b58b52ee7db" candidate_uuid: "89af51f328514c008db5fb2f58b6c233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964112d582014c439911bbfe000fd105" is_pre_election: true
I20260810 15:19:48.574503 31695 raft_consensus.cc:2454] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af51f328514c008db5fb2f58b6c233 in term 0.
I20260810 15:19:48.574502 31470 raft_consensus.cc:2454] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89af51f328514c008db5fb2f58b6c233 in term 0.
I20260810 15:19:48.574707 31818 leader_election.cc:302] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89af51f328514c008db5fb2f58b6c233, 964112d582014c439911bbfe000fd105; no voters:
I20260810 15:19:48.574823 32050 raft_consensus.cc:2781] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:48.574887 32050 raft_consensus.cc:482] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:48.574916 32050 raft_consensus.cc:3037] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:48.575670 32050 raft_consensus.cc:504] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.575788 32050 leader_election.cc:288] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [CANDIDATE]: Term 1 election: Requested vote from peers 3f30ee1a2f3940be8f35612c1d9e48b4 (127.29.230.2:38829), 964112d582014c439911bbfe000fd105 (127.29.230.1:41457)
I20260810 15:19:48.575944 31695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f643a3ef2e5f45a38e675b58b52ee7db" candidate_uuid: "89af51f328514c008db5fb2f58b6c233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4"
I20260810 15:19:48.575978 31470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f643a3ef2e5f45a38e675b58b52ee7db" candidate_uuid: "89af51f328514c008db5fb2f58b6c233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964112d582014c439911bbfe000fd105"
I20260810 15:19:48.576022 31695 raft_consensus.cc:3037] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:48.576050 31470 raft_consensus.cc:3037] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:48.576768 31470 raft_consensus.cc:2454] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af51f328514c008db5fb2f58b6c233 in term 1.
I20260810 15:19:48.576807 31695 raft_consensus.cc:2454] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89af51f328514c008db5fb2f58b6c233 in term 1.
I20260810 15:19:48.576962 31816 leader_election.cc:302] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f30ee1a2f3940be8f35612c1d9e48b4, 89af51f328514c008db5fb2f58b6c233; no voters:
I20260810 15:19:48.577060 32050 raft_consensus.cc:2781] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:48.577199 32050 raft_consensus.cc:686] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 1 LEADER]: Becoming Leader. State: Replica: 89af51f328514c008db5fb2f58b6c233, State: Running, Role: LEADER
I20260810 15:19:48.577279 32050 consensus_queue.cc:237] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } }
I20260810 15:19:48.578078 31227 catalog_manager.cc:5495] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89af51f328514c008db5fb2f58b6c233 (127.29.230.3). New cstate: current_term: 1 leader_uuid: "89af51f328514c008db5fb2f58b6c233" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89af51f328514c008db5fb2f58b6c233" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 44375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 } health_report { overall_health: UNKNOWN } } }
I20260810 15:19:48.606688 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.604536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.29.230.3:44375)
0810 15:19:48.604589 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:48.604591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:48.604616 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:19:48.604647 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:48.604648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:48.604667 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:48.604719 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:48.604831 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.605295 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.606046 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:48.606239 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:48.606294 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:48.606401 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:48.606496 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:48.606563 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:48.606605 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:19:48.606612 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:19:48.608587 31695 raft_consensus.cc:1261] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 1 FOLLOWER]: Refusing update from remote peer 89af51f328514c008db5fb2f58b6c233: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:48.608587 31470 raft_consensus.cc:1261] T f643a3ef2e5f45a38e675b58b52ee7db P 964112d582014c439911bbfe000fd105 [term 1 FOLLOWER]: Refusing update from remote peer 89af51f328514c008db5fb2f58b6c233: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:48.608971 32055 consensus_queue.cc:1035] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f30ee1a2f3940be8f35612c1d9e48b4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:48.609098 32050 consensus_queue.cc:1035] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [LEADER]: Connected to new peer: Peer: permanent_uuid: "964112d582014c439911bbfe000fd105" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 41457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:48.611321 32059 mvcc.cc:205] Tried to move back new op lower bound from 7316992772538060800 to 7316992772413337600. Current Snapshot: MvccSnapshot[applied={T|T < 7316992772538060800}]
I20260810 15:19:48.612203 32061 mvcc.cc:205] Tried to move back new op lower bound from 7316992772538060800 to 7316992772413337600. Current Snapshot: MvccSnapshot[applied={T|T < 7316992772538060800}]
I20260810 15:19:48.613951 31231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:48.614060 31231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:48.614776 31231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1c1eb778a574bcd9f2d64bc54dc6108: Sending DeleteTablet for 3 replicas of tablet f643a3ef2e5f45a38e675b58b52ee7db
I20260810 15:19:48.615113 31898 tablet_service.cc:1514] Processing DeleteTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:48 UTC) from {username='slave'} at 127.0.0.1:45440
I20260810 15:19:48.615128 31658 tablet_service.cc:1514] Processing DeleteTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:48 UTC) from {username='slave'} at 127.0.0.1:51780
I20260810 15:19:48.615207 31438 tablet_service.cc:1514] Processing DeleteTablet for tablet f643a3ef2e5f45a38e675b58b52ee7db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:48 UTC) from {username='slave'} at 127.0.0.1:50052
I20260810 15:19:48.615437 32078 tablet_replica.cc:330] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: stopping tablet replica
I20260810 15:19:48.615530 32078 raft_consensus.cc:2229] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:48.615615 32078 raft_consensus.cc:2258] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:48.615746 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31325
I20260810 15:19:48.629339 32078 ts_tablet_manager.cc:1905] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:48.630788 32078 ts_tablet_manager.cc:1918] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:48.630869 32078 log.cc:1196] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/f643a3ef2e5f45a38e675b58b52ee7db
I20260810 15:19:48.631085 32078 ts_tablet_manager.cc:1939] T f643a3ef2e5f45a38e675b58b52ee7db P 3f30ee1a2f3940be8f35612c1d9e48b4: Deleting consensus metadata
I20260810 15:19:48.631615 31200 catalog_manager.cc:4841] TS 3f30ee1a2f3940be8f35612c1d9e48b4 (127.29.230.2:38829): tablet f643a3ef2e5f45a38e675b58b52ee7db (table test-table [id=831a1a1a5adf433f925ac89ab4e1ced8]) successfully deleted
W20260810 15:19:48.634968 31203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:41457 (error 108)
I20260810 15:19:48.635291 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31556
I20260810 15:19:48.635470 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:48.635335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:41457 (local address 127.0.0.1:50058)
0810 15:19:48.635386 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:48.635436 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:41457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:19:48.635578 31203 catalog_manager.cc:4584] TS 964112d582014c439911bbfe000fd105 (127.29.230.1:41457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f643a3ef2e5f45a38e675b58b52ee7db: Network error: Client connection negotiation failed: client connection to 127.29.230.1:41457: connect: Connection refused (error 111)
I20260810 15:19:48.635712 32077 tablet_replica.cc:330] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: stopping tablet replica
I20260810 15:19:48.635818 32077 raft_consensus.cc:2229] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:48.635929 32077 raft_consensus.cc:2258] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:48.649466 32077 ts_tablet_manager.cc:1905] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:48.652078 32077 ts_tablet_manager.cc:1918] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:48.652163 32077 log.cc:1196] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/f643a3ef2e5f45a38e675b58b52ee7db
I20260810 15:19:48.652406 32077 ts_tablet_manager.cc:1939] T f643a3ef2e5f45a38e675b58b52ee7db P 89af51f328514c008db5fb2f58b6c233: Deleting consensus metadata
I20260810 15:19:48.652945 31200 catalog_manager.cc:4841] TS 89af51f328514c008db5fb2f58b6c233 (127.29.230.3:44375): tablet f643a3ef2e5f45a38e675b58b52ee7db (table test-table [id=831a1a1a5adf433f925ac89ab4e1ced8]) successfully deleted
I20260810 15:19:48.657743 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31789
I20260810 15:19:48.670660 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 31176
2026-08-10T15:19:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (826 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:48 dist-test-slave-gjrz krb5kdc[32090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:48 dist-test-slave-gjrz krb5kdc[32090](info): set up 2 sockets
Aug 10 15:19:48 dist-test-slave-gjrz krb5kdc[32090](info): commencing operation
krb5kdc: starting...
W20260810 15:19:49.834105 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:49 dist-test-slave-gjrz krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:19:49.954744 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46445
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:46445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:50.037531 32115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.037731 32115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.037753 32115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.038854 32115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:50.038892 32115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.038904 32115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.038916 32115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:50.038926 32115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:50.040465 32115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:46445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:46445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.040668 32115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.040843 32115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:50.045655 32124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.045642 32127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.045756 32115 server_base.cc:1034] running on GCE node
W20260810 15:19:50.045674 32121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.046054 32115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.046330 32115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.047509 32115 hybrid_clock.cc:648] HybridClock initialized: now 1786375190047493 us; error 30 us; skew 500 ppm
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:50.049993 32115 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:19:50.050467 32115 webserver.cc:460] Webserver started at http://127.29.230.62:41847/ using document root <none> and password file <none>
I20260810 15:19:50.050669 32115 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.050712 32115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.050834 32115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.051925 32115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "69373dc523a14561b351e663fe09e0b6"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "c9ed7957293eadc33e66a1ad8be2ac64"
server_key_iv: "7daa23b39bcd19a7379f47c21bac23ae"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.052253 32115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "69373dc523a14561b351e663fe09e0b6"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "c9ed7957293eadc33e66a1ad8be2ac64"
server_key_iv: "7daa23b39bcd19a7379f47c21bac23ae"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.053509 32115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:50.054177 32138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.054327 32115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:50.054416 32115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "69373dc523a14561b351e663fe09e0b6"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "c9ed7957293eadc33e66a1ad8be2ac64"
server_key_iv: "7daa23b39bcd19a7379f47c21bac23ae"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.054482 32115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:50.067992 32115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:50.069980 32115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:50.070119 32115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:50.074015 32115 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:46445
I20260810 15:19:50.074064 32224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:46445 every 8 connection(s)
I20260810 15:19:50.074375 32115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:50.075227 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:50.077527 32229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: Bootstrap starting.
I20260810 15:19:50.078083 32229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:50.078311 32229 log.cc:824] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:50.078950 32229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: No bootstrap required, opened a new log
I20260810 15:19:50.079903 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32115
I20260810 15:19:50.080017 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:50.080212 30616 external_mini_cluster.cc:1428] Setting key e3c7537d031487e9144c8b87a1c8864e
I20260810 15:19:50.081001 32229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } }
I20260810 15:19:50.081147 32229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:50.081167 32229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69373dc523a14561b351e663fe09e0b6, State: Initialized, Role: FOLLOWER
I20260810 15:19:50.081270 32229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } }
I20260810 15:19:50.081352 32229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:50.081382 32229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:50.081431 32229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:50.082407 32229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } }
I20260810 15:19:50.082497 32229 leader_election.cc:302] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69373dc523a14561b351e663fe09e0b6; no voters:
I20260810 15:19:50.082647 32229 leader_election.cc:288] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:50.082696 32235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:50.082840 32235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 69373dc523a14561b351e663fe09e0b6, State: Running, Role: LEADER
I20260810 15:19:50.082895 32229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:50.082928 32235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } }
I20260810 15:19:50.083257 32238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69373dc523a14561b351e663fe09e0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } } }
I20260810 15:19:50.083292 32239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69373dc523a14561b351e663fe09e0b6. Latest consensus state: current_term: 1 leader_uuid: "69373dc523a14561b351e663fe09e0b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69373dc523a14561b351e663fe09e0b6" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 46445 } } }
I20260810 15:19:50.083390 32238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:50.083402 32239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:50.083600 32248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:50.084139 32248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:50.085742 32248 catalog_manager.cc:1347] Generated new cluster ID: fef730715f4247e99f21bc5449b6a2a4
I20260810 15:19:50.085790 32248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:50.091502 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.080727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.29.230.62:46445)
0810 15:19:50.080894 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.080902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.080931 (+ 29us) server_negotiation.cc:407] Connection header received
0810 15:19:50.081064 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.081069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.081111 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.081246 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.082083 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.083310 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.084448 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.085046 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.085186 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.087878 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.087882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.087883 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:50.087897 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.089815 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.090205 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.090208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.090209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.090346 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.090537 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.090538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.090539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.090880 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.091058 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.091127 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.091300 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260810 15:19:50.101464 32248 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:50.102049 32248 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:50.104589 32248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: Generated new TSK 0
I20260810 15:19:50.104753 32248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.230.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
Entry for principal kudu/127.29.230.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab.
I20260810 15:19:50.132443 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:50.213995 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.214196 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.214227 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.215353 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.215392 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.215425 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:50.216828 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.1.keytab
--principal=kudu/127.29.230.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.217074 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.217276 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:50.222216 32285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.222244 32282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.222445 32276 server_base.cc:1034] running on GCE node
W20260810 15:19:50.222254 32287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.222733 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.222993 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.224135 32276 hybrid_clock.cc:648] HybridClock initialized: now 1786375190224112 us; error 38 us; skew 500 ppm
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:50.226606 32276 init.cc:377] Logged in from keytab as kudu/127.29.230.1@KRBTEST.COM (short username kudu)
I20260810 15:19:50.227010 32276 webserver.cc:460] Webserver started at http://127.29.230.1:43495/ using document root <none> and password file <none>
I20260810 15:19:50.227210 32276 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.227257 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.227425 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.228461 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "f462b72f06b142ecb6fd8fab47ec059f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "67ba79c15811ba419ebd8c0228a9aa42"
server_key_iv: "3a2039249575b66d6257abd123263415"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.228801 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "f462b72f06b142ecb6fd8fab47ec059f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "67ba79c15811ba419ebd8c0228a9aa42"
server_key_iv: "3a2039249575b66d6257abd123263415"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.230039 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:50.230726 32300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.230871 32276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:50.230957 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "f462b72f06b142ecb6fd8fab47ec059f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "67ba79c15811ba419ebd8c0228a9aa42"
server_key_iv: "3a2039249575b66d6257abd123263415"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.231026 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:50.247455 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:50.249478 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:50.249630 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:50.249856 32276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:50.250178 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:50.250216 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.250243 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:50.250265 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.255858 32276 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:43033
I20260810 15:19:50.255904 32474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:43033 every 8 connection(s)
I20260810 15:19:50.256218 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:50.258235 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32276
I20260810 15:19:50.258351 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:50.258548 30616 external_mini_cluster.cc:1428] Setting key 4d9053eb723b906bb497a62802838068
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.1@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:50.266479 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.256945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:57597 (local address 127.29.230.62:46445)
0810 15:19:50.256993 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.256995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.257338 (+ 343us) server_negotiation.cc:407] Connection header received
0810 15:19:50.258099 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.258100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.258119 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.258196 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.259330 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.259994 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.261000 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.261216 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.261273 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.263600 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.263602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.263602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:50.263613 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.265151 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.265573 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.265574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.265574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.265639 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.265845 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.265846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.265846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.265995 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.266065 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.266294 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.266386 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:19:50.266584 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.256905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46445 (local address 127.29.230.1:57597)
0810 15:19:50.257213 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.257257 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.257917 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.258270 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.258273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.258978 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:50.259204 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.259211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.260077 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.260078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.260916 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.260919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.261288 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.261289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.261439 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.261928 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:50.261939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:50.263488 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:50.265252 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.265259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.265261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.265502 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.265701 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.265702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.265702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.265794 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.266052 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:50.266055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:50.266176 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.266257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.266381 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260810 15:19:50.267112 32475 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46445
I20260810 15:19:50.267223 32475 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:50.267449 32475 heartbeater.cc:511] Master 127.29.230.62:46445 requested a full tablet report, sending...
I20260810 15:19:50.268106 32162 ts_manager.cc:194] Registered new tserver with Master: f462b72f06b142ecb6fd8fab47ec059f (127.29.230.1:43033)
WARNING: no policy specified for kudu/127.29.230.2@KRBTEST.COM; defaulting to no policy
I20260810 15:19:50.269037 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.1@KRBTEST.COM'} at 127.29.230.1:57597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
Entry for principal kudu/127.29.230.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab.
I20260810 15:19:50.286262 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:50.368131 32484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.368338 32484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.368360 32484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.369482 32484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.369526 32484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.369551 32484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:50.370982 32484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.2.keytab
--principal=kudu/127.29.230.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.371210 32484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.371445 32484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:50.376196 32512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.376206 32507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.376214 32506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.376464 32484 server_base.cc:1034] running on GCE node
I20260810 15:19:50.376658 32484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.376894 32484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.378052 32484 hybrid_clock.cc:648] HybridClock initialized: now 1786375190378040 us; error 27 us; skew 500 ppm
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:50.380426 32484 init.cc:377] Logged in from keytab as kudu/127.29.230.2@KRBTEST.COM (short username kudu)
I20260810 15:19:50.380836 32484 webserver.cc:460] Webserver started at http://127.29.230.2:45001/ using document root <none> and password file <none>
I20260810 15:19:50.381062 32484 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.381105 32484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.381196 32484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.382181 32484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "a6f781c7fde941ee8b5fcbcf40a157c4"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "47f8c60ed7f14553a490f6eb5e384f5e"
server_key_iv: "3f7ccca2ee231c65326c98e3521f388b"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.382470 32484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "a6f781c7fde941ee8b5fcbcf40a157c4"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "47f8c60ed7f14553a490f6eb5e384f5e"
server_key_iv: "3f7ccca2ee231c65326c98e3521f388b"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.383844 32484 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:50.384626 32529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.384779 32484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:50.384881 32484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "a6f781c7fde941ee8b5fcbcf40a157c4"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "47f8c60ed7f14553a490f6eb5e384f5e"
server_key_iv: "3f7ccca2ee231c65326c98e3521f388b"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.384949 32484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:50.391882 32484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:50.393656 32484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:50.393787 32484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:50.393981 32484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:50.394250 32484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:50.394282 32484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.394309 32484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:50.394332 32484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.399868 32484 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:37299
I20260810 15:19:50.399959 32721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:37299 every 8 connection(s)
I20260810 15:19:50.400228 32484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:50.402133 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32484
I20260810 15:19:50.402249 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:50.402434 30616 external_mini_cluster.cc:1428] Setting key 6dd2ec24fddb6f798ebadcc174126574
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.2@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:50.410583 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.400924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:41395 (local address 127.29.230.62:46445)
0810 15:19:50.401003 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.401005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.401177 (+ 172us) server_negotiation.cc:407] Connection header received
0810 15:19:50.401673 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.401674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.401685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.401733 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.402905 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.403566 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.404907 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.405161 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.405241 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.407455 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.407456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.407457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:50.407465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.409121 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.409547 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.409548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.409548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.409622 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.409837 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.409838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.409838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.410005 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.410080 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.410381 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.410482 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:19:50.410764 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.400887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46445 (local address 127.29.230.2:41395)
0810 15:19:50.401054 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.401092 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.401580 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.401795 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.401798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.402561 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:50.402816 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.402823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.403654 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.403656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.404808 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.404812 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.405253 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.405254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.405360 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.405864 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:50.405872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:50.407361 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:50.409221 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.409228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.409230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.409470 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.409683 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.409684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.409684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.409782 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.410098 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:50.410104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:50.410246 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.410344 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.410481 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260810 15:19:50.411339 32722 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46445
I20260810 15:19:50.411444 32722 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:50.411631 32722 heartbeater.cc:511] Master 127.29.230.62:46445 requested a full tablet report, sending...
I20260810 15:19:50.412039 32162 ts_manager.cc:194] Registered new tserver with Master: a6f781c7fde941ee8b5fcbcf40a157c4 (127.29.230.2:37299)
I20260810 15:19:50.412550 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.2@KRBTEST.COM'} at 127.29.230.2:41395
WARNING: no policy specified for kudu/127.29.230.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
Entry for principal kudu/127.29.230.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab.
I20260810 15:19:50.430673 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:50.512053 32727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.512276 32727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.512300 32727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.513458 32727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.513506 32727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.513532 32727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:50.514998 32727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.3.keytab
--principal=kudu/127.29.230.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:46445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.515237 32727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.515471 32727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:50.520467 32738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.520444 32732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.520444 32741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.520670 32727 server_base.cc:1034] running on GCE node
I20260810 15:19:50.520925 32727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.521140 32727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.522300 32727 hybrid_clock.cc:648] HybridClock initialized: now 1786375190522282 us; error 31 us; skew 500 ppm
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:50.524914 32727 init.cc:377] Logged in from keytab as kudu/127.29.230.3@KRBTEST.COM (short username kudu)
I20260810 15:19:50.525447 32727 webserver.cc:460] Webserver started at http://127.29.230.3:34445/ using document root <none> and password file <none>
I20260810 15:19:50.525681 32727 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.525745 32727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.525877 32727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.526902 32727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "4c60cf6006d7441c92840b098388be6f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "75afdb4601efc2570a412676d3f88f5b"
server_key_iv: "f4256de91b7713e1f61353cf70e1b5ec"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.527244 32727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "4c60cf6006d7441c92840b098388be6f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "75afdb4601efc2570a412676d3f88f5b"
server_key_iv: "f4256de91b7713e1f61353cf70e1b5ec"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.528564 32727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:50.529304 32752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.529457 32727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:50.529544 32727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "4c60cf6006d7441c92840b098388be6f"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "75afdb4601efc2570a412676d3f88f5b"
server_key_iv: "f4256de91b7713e1f61353cf70e1b5ec"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.529615 32727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:50.545728 32727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:50.547786 32727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:50.547915 32727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:50.548179 32727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:50.548503 32727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:50.548538 32727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.548561 32727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:50.548580 32727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.554387 32727 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:38045
I20260810 15:19:50.554438 472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:38045 every 8 connection(s)
I20260810 15:19:50.554752 32727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:50.555912 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 32727
I20260810 15:19:50.556025 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:50.556208 30616 external_mini_cluster.cc:1428] Setting key 5f85f16c2bc5e87d206b0c5cf9d2a571
Aug 10 15:19:50 dist-test-slave-gjrz krb5kdc[32090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375190, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.3@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:19:50.564298 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.555549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:59943 (local address 127.29.230.62:46445)
0810 15:19:50.555610 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.555612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.555842 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:19:50.556448 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.556449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.556463 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.556539 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.557649 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.558252 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.559224 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.559437 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.559492 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.561612 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.561614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.561614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:50.561621 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.562985 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.563395 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.563395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.563396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.563466 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.563662 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.563663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.563663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.563758 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.563802 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.564157 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.564231 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:19:50.564435 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.555519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:46445 (local address 127.29.230.3:59943)
0810 15:19:50.555689 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.555752 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.556333 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.556685 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.556690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.557364 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:50.557573 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.557578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.558338 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.558339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.559148 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.559151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.559559 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.559560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.559684 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.560162 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:19:50.560171 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:50.561514 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:50.563070 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.563076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.563078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.563338 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.563514 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:19:50.563515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:19:50.563515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:19:50.563620 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:19:50.563834 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:50.563840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:50.563967 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.564104 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.564243 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260810 15:19:50.564917 473 heartbeater.cc:347] Connected to a master server at 127.29.230.62:46445
I20260810 15:19:50.565052 473 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:50.565232 473 heartbeater.cc:511] Master 127.29.230.62:46445 requested a full tablet report, sending...
I20260810 15:19:50.565671 32162 ts_manager.cc:194] Registered new tserver with Master: 4c60cf6006d7441c92840b098388be6f (127.29.230.3:38045)
I20260810 15:19:50.566035 32162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.230.3@KRBTEST.COM'} at 127.29.230.3:59943
I20260810 15:19:50.567880 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:50.574669 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.569162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.29.230.62:46445)
0810 15:19:50.569220 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.569222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.569307 (+ 85us) server_negotiation.cc:407] Connection header received
0810 15:19:50.569366 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.569366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.569376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.569418 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.569944 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.570405 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.571184 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.571428 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.571476 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.572503 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.572504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.572505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:50.572512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.573642 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.573938 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.573939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.573939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.573994 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:50.574157 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:50.574158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:50.574158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:50.574301 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.574386 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.574509 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.574583 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:19:50.576094 32162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:50.577598 32162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:50.583859 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:37299 (local address 127.0.0.1:34806)
0810 15:19:50.579812 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.579854 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.579921 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.580178 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.580182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.580184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:50.580250 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.580252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581647 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.581648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.582625 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.582628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583567 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.583645 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583662 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.583728 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.583782 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":10,"thread_start_us":57,"threads_started":1}
I20260810 15:19:50.583985 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:43033 (local address 127.0.0.1:47932)
0810 15:19:50.580054 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.580063 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.580150 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.580427 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.580428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.580430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:50.580481 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.580482 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581456 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.581456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.582696 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.582699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583705 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.583764 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583769 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.583815 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.583860 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260810 15:19:50.584029 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.29.230.1:43033)
0810 15:19:50.580002 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.580007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.580123 (+ 116us) server_negotiation.cc:407] Connection header received
0810 15:19:50.580238 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.580241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.580274 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.580325 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:50.580634 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581360 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.582794 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583625 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583671 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583726 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.583911 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.583957 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260810 15:19:50.584203 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.29.230.2:37299)
0810 15:19:50.580002 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.580006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.580024 (+ 18us) server_negotiation.cc:407] Connection header received
0810 15:19:50.580065 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.580067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.580084 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.580163 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:50.580338 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581534 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.582699 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583450 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583527 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583630 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.584075 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.584121 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":83,"threads_started":1}
I20260810 15:19:50.584685 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:38045 (local address 127.0.0.1:54082)
0810 15:19:50.579827 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.579857 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.579921 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.580288 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.580289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.580290 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:50.580323 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.580324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581639 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.581640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.582799 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.582801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583825 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.583883 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583887 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.584566 (+ 679us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.584613 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":25,"thread_start_us":61,"threads_started":1}
I20260810 15:19:50.584712 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.579789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.29.230.3:38045)
0810 15:19:50.580087 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.580091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.580100 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:50.580128 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.580130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.580144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.580217 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:50.580393 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.581529 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.582881 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.583748 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.583809 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.583862 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.584599 (+ 737us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.584646 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":86,"threads_started":1}
I20260810 15:19:50.584764 32379 tablet_service.cc:1467] Processing CreateTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e (DEFAULT_TABLE table=test-table [id=74c95b71d182476ab135f916c6ca050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:50.584920 32612 tablet_service.cc:1467] Processing CreateTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e (DEFAULT_TABLE table=test-table [id=74c95b71d182476ab135f916c6ca050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:50.585184 32379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803832b46ed84d9cbb07905c4fc73c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:50.585338 370 tablet_service.cc:1467] Processing CreateTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e (DEFAULT_TABLE table=test-table [id=74c95b71d182476ab135f916c6ca050a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:50.585481 32612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803832b46ed84d9cbb07905c4fc73c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:50.585676 370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803832b46ed84d9cbb07905c4fc73c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:50.588768 505 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Bootstrap starting.
I20260810 15:19:50.589380 505 tablet_bootstrap.cc:654] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:50.589823 505 log.cc:824] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:50.590132 507 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: Bootstrap starting.
I20260810 15:19:50.590485 505 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: No bootstrap required, opened a new log
I20260810 15:19:50.590555 505 ts_tablet_manager.cc:1397] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:50.590678 507 tablet_bootstrap.cc:654] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:50.591130 507 log.cc:824] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:50.591420 506 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Bootstrap starting.
I20260810 15:19:50.591835 507 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: No bootstrap required, opened a new log
I20260810 15:19:50.591902 507 ts_tablet_manager.cc:1397] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 15:19:50.592119 506 tablet_bootstrap.cc:654] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:50.592705 506 log.cc:824] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:50.592700 505 raft_consensus.cc:348] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.592900 505 raft_consensus.cc:374] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:50.593010 505 raft_consensus.cc:729] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c60cf6006d7441c92840b098388be6f, State: Initialized, Role: FOLLOWER
I20260810 15:19:50.593132 505 consensus_queue.cc:260] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.593393 505 ts_tablet_manager.cc:1428] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:50.593426 473 heartbeater.cc:503] Master 127.29.230.62:46445 was elected leader, sending a full tablet report...
I20260810 15:19:50.593590 507 raft_consensus.cc:348] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.593775 507 raft_consensus.cc:374] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:50.593823 507 raft_consensus.cc:729] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f462b72f06b142ecb6fd8fab47ec059f, State: Initialized, Role: FOLLOWER
I20260810 15:19:50.593932 507 consensus_queue.cc:260] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.594097 506 tablet_bootstrap.cc:492] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: No bootstrap required, opened a new log
I20260810 15:19:50.594158 506 ts_tablet_manager.cc:1397] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 15:19:50.594201 507 ts_tablet_manager.cc:1428] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 15:19:50.594211 32475 heartbeater.cc:503] Master 127.29.230.62:46445 was elected leader, sending a full tablet report...
I20260810 15:19:50.595983 506 raft_consensus.cc:348] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.596120 506 raft_consensus.cc:374] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:50.596151 506 raft_consensus.cc:729] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f781c7fde941ee8b5fcbcf40a157c4, State: Initialized, Role: FOLLOWER
I20260810 15:19:50.596261 506 consensus_queue.cc:260] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.596508 506 ts_tablet_manager.cc:1428] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 15:19:50.596549 32722 heartbeater.cc:503] Master 127.29.230.62:46445 was elected leader, sending a full tablet report...
I20260810 15:19:50.632640 518 raft_consensus.cc:482] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:50.632787 518 raft_consensus.cc:504] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.633224 518 leader_election.cc:288] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f462b72f06b142ecb6fd8fab47ec059f (127.29.230.1:43033), 4c60cf6006d7441c92840b098388be6f (127.29.230.3:38045)
I20260810 15:19:50.637156 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.633627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:53835 (local address 127.29.230.1:43033)
0810 15:19:50.633682 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.633684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.633693 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:50.633718 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.633719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.633728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.633772 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:50.633896 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.634622 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636194 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.636783 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636861 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.636870 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.637038 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.637089 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:19:50.637156 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.633531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:52457 (local address 127.29.230.3:38045)
0810 15:19:50.633576 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.633577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.633588 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:19:50.633616 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.633617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.633626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.633684 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:50.633843 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.634648 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636194 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.636782 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636861 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.636870 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.637038 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.637081 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:19:50.637176 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.633404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:38045 (local address 127.29.230.2:52457)
0810 15:19:50.633460 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.633472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.633512 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.633699 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.633700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.633701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:50.633774 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.633777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.634740 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.634741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.636091 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.636094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.636858 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.636969 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.636971 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.637019 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.637071 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":10}
I20260810 15:19:50.637184 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.633405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:43033 (local address 127.29.230.2:53835)
0810 15:19:50.633605 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.633615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:50.633661 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:50.633784 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:50.633784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:50.633785 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:50.633838 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.633839 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.634739 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.634746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.636091 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:50.636094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.636858 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.636859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:50.636954 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.636959 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:19:50.637017 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 15:19:50.637071 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":97,"thread_start_us":118,"threads_started":1}
I20260810 15:19:50.637554 406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803832b46ed84d9cbb07905c4fc73c8e" candidate_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60cf6006d7441c92840b098388be6f" is_pre_election: true
I20260810 15:19:50.637720 406 raft_consensus.cc:2454] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f781c7fde941ee8b5fcbcf40a157c4 in term 0.
I20260810 15:19:50.637699 32409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803832b46ed84d9cbb07905c4fc73c8e" candidate_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f462b72f06b142ecb6fd8fab47ec059f" is_pre_election: true
I20260810 15:19:50.637844 32409 raft_consensus.cc:2454] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6f781c7fde941ee8b5fcbcf40a157c4 in term 0.
I20260810 15:19:50.637955 32530 leader_election.cc:302] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c60cf6006d7441c92840b098388be6f, a6f781c7fde941ee8b5fcbcf40a157c4; no voters:
I20260810 15:19:50.638079 518 raft_consensus.cc:2781] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:50.638135 518 raft_consensus.cc:482] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:50.638157 518 raft_consensus.cc:3037] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:50.638876 518 raft_consensus.cc:504] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.639004 518 leader_election.cc:288] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [CANDIDATE]: Term 1 election: Requested vote from peers f462b72f06b142ecb6fd8fab47ec059f (127.29.230.1:43033), 4c60cf6006d7441c92840b098388be6f (127.29.230.3:38045)
I20260810 15:19:50.639164 32409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803832b46ed84d9cbb07905c4fc73c8e" candidate_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f462b72f06b142ecb6fd8fab47ec059f"
I20260810 15:19:50.639235 32409 raft_consensus.cc:3037] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:50.639217 406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803832b46ed84d9cbb07905c4fc73c8e" candidate_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c60cf6006d7441c92840b098388be6f"
I20260810 15:19:50.639322 406 raft_consensus.cc:3037] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:50.639966 32409 raft_consensus.cc:2454] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f781c7fde941ee8b5fcbcf40a157c4 in term 1.
I20260810 15:19:50.640005 406 raft_consensus.cc:2454] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6f781c7fde941ee8b5fcbcf40a157c4 in term 1.
I20260810 15:19:50.640175 32532 leader_election.cc:302] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6f781c7fde941ee8b5fcbcf40a157c4, f462b72f06b142ecb6fd8fab47ec059f; no voters:
I20260810 15:19:50.640261 518 raft_consensus.cc:2781] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:50.640435 518 raft_consensus.cc:686] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 1 LEADER]: Becoming Leader. State: Replica: a6f781c7fde941ee8b5fcbcf40a157c4, State: Running, Role: LEADER
I20260810 15:19:50.640542 518 consensus_queue.cc:237] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } }
I20260810 15:19:50.641361 32160 catalog_manager.cc:5495] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6f781c7fde941ee8b5fcbcf40a157c4 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6f781c7fde941ee8b5fcbcf40a157c4" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 37299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 } health_report { overall_health: UNKNOWN } } }
W20260810 15:19:50.650729 32723 tablet.cc:2369] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:19:50.660555 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.658172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.29.230.2:37299)
0810 15:19:50.658226 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.658228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.658250 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:19:50.658278 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.658279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.658294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.658348 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:50.658526 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.658983 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.659811 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.660016 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.660074 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.660280 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:50.660381 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:50.660458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.660474 (+ 16us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.660495 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 15:19:50.662266 406 raft_consensus.cc:1261] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 1 FOLLOWER]: Refusing update from remote peer a6f781c7fde941ee8b5fcbcf40a157c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:50.662271 32409 raft_consensus.cc:1261] T 803832b46ed84d9cbb07905c4fc73c8e P f462b72f06b142ecb6fd8fab47ec059f [term 1 FOLLOWER]: Refusing update from remote peer a6f781c7fde941ee8b5fcbcf40a157c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:50.662643 520 consensus_queue.cc:1035] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c60cf6006d7441c92840b098388be6f" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:50.662786 518 consensus_queue.cc:1035] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f462b72f06b142ecb6fd8fab47ec059f" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 43033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:50.665084 524 mvcc.cc:205] Tried to move back new op lower bound from 7316992780950175744 to 7316992780864270336. Current Snapshot: MvccSnapshot[applied={T|T < 7316992780950175744}]
I20260810 15:19:50.665472 525 mvcc.cc:205] Tried to move back new op lower bound from 7316992780950175744 to 7316992780864270336. Current Snapshot: MvccSnapshot[applied={T|T < 7316992780950175744 or (T in {7316992780950175744})}]
I20260810 15:19:50.665663 527 mvcc.cc:205] Tried to move back new op lower bound from 7316992780950175744 to 7316992780864270336. Current Snapshot: MvccSnapshot[applied={T|T < 7316992780950175744}]
I20260810 15:19:50.667841 32160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:50.667924 32160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:50.668659 32160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69373dc523a14561b351e663fe09e0b6: Sending DeleteTablet for 3 replicas of tablet 803832b46ed84d9cbb07905c4fc73c8e
I20260810 15:19:50.668989 32379 tablet_service.cc:1514] Processing DeleteTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:50 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:47932
I20260810 15:19:50.669006 370 tablet_service.cc:1514] Processing DeleteTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:50 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:54082
I20260810 15:19:50.669206 32612 tablet_service.cc:1514] Processing DeleteTablet for tablet 803832b46ed84d9cbb07905c4fc73c8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:50 UTC) from {username='kudu', principal='kudu/127.29.230.62@KRBTEST.COM'} at 127.0.0.1:34806
I20260810 15:19:50.669250 547 tablet_replica.cc:330] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: stopping tablet replica
I20260810 15:19:50.669423 547 raft_consensus.cc:2229] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:50.669451 548 tablet_replica.cc:330] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: stopping tablet replica
I20260810 15:19:50.669552 548 raft_consensus.cc:2229] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:50.669644 547 raft_consensus.cc:2258] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:50.669674 548 raft_consensus.cc:2258] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:50.669673 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32276
I20260810 15:19:50.670480 547 ts_tablet_manager.cc:1905] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:50.672418 547 ts_tablet_manager.cc:1918] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:50.672753 547 log.cc:1196] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/803832b46ed84d9cbb07905c4fc73c8e
I20260810 15:19:50.673167 547 ts_tablet_manager.cc:1939] T 803832b46ed84d9cbb07905c4fc73c8e P 4c60cf6006d7441c92840b098388be6f: Deleting consensus metadata
I20260810 15:19:50.673688 32139 catalog_manager.cc:4841] TS 4c60cf6006d7441c92840b098388be6f (127.29.230.3:38045): tablet 803832b46ed84d9cbb07905c4fc73c8e (table test-table [id=74c95b71d182476ab135f916c6ca050a]) successfully deleted
I20260810 15:19:50.677596 548 ts_tablet_manager.cc:1905] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:50.679354 548 ts_tablet_manager.cc:1918] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:50.679450 548 log.cc:1196] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/803832b46ed84d9cbb07905c4fc73c8e
I20260810 15:19:50.679735 548 ts_tablet_manager.cc:1939] T 803832b46ed84d9cbb07905c4fc73c8e P a6f781c7fde941ee8b5fcbcf40a157c4: Deleting consensus metadata
I20260810 15:19:50.680212 32140 catalog_manager.cc:4841] TS a6f781c7fde941ee8b5fcbcf40a157c4 (127.29.230.2:37299): tablet 803832b46ed84d9cbb07905c4fc73c8e (table test-table [id=74c95b71d182476ab135f916c6ca050a]) successfully deleted
W20260810 15:19:50.690358 32143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:43033 (error 108)
I20260810 15:19:50.690591 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32484
I20260810 15:19:50.690930 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.690800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:43033 (local address 127.0.0.1:47942)
0810 15:19:50.690851 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:50.690883 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 15:19:50.691097 32143 catalog_manager.cc:4584] TS f462b72f06b142ecb6fd8fab47ec059f (127.29.230.1:43033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 803832b46ed84d9cbb07905c4fc73c8e: Network error: Client connection negotiation failed: client connection to 127.29.230.1:43033: connect: Connection refused (error 111)
I20260810 15:19:50.705593 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32727
I20260810 15:19:50.716245 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 32115
2026-08-10T15:19:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2045 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T15:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:50Z Disabled control of system clock
I20260810 15:19:50.746929 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33905
--webserver_interface=127.29.230.62
--webserver_port=0
--builtin_ntp_servers=127.29.230.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:33905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:50.828568 557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.828776 557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.828794 557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.829880 557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:50.829913 557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.829924 557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.829936 557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:50.829954 557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:50.831305 557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:33905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.831493 557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.831738 557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:19:50.836465 557 server_base.cc:1034] running on GCE node
W20260810 15:19:50.836395 567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.836401 565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.836400 571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.836917 557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.837227 557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.838382 557 hybrid_clock.cc:648] HybridClock initialized: now 1786375190838365 us; error 27 us; skew 500 ppm
I20260810 15:19:50.839644 557 webserver.cc:460] Webserver started at http://127.29.230.62:35141/ using document root <none> and password file <none>
I20260810 15:19:50.839879 557 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.839929 557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.840067 557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.841100 557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "6a12c691fa5b48beba7eae13d47613b7"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "e45bb10461716fdb00f67975a09e0d5d"
server_key_iv: "3a02553073db1fe306db8c5f2d0a90c3"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.841467 557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "6a12c691fa5b48beba7eae13d47613b7"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "e45bb10461716fdb00f67975a09e0d5d"
server_key_iv: "3a02553073db1fe306db8c5f2d0a90c3"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.842691 557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:50.843393 579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:50.843549 557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:50.843636 557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "6a12c691fa5b48beba7eae13d47613b7"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "e45bb10461716fdb00f67975a09e0d5d"
server_key_iv: "3a02553073db1fe306db8c5f2d0a90c3"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.843703 557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:50.864392 557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:50.864749 557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:50.864905 557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:50.868933 557 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:33905
I20260810 15:19:50.868973 672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:33905 every 8 connection(s)
I20260810 15:19:50.869307 557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:50.870303 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:50.871666 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 557
I20260810 15:19:50.871752 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:50.871919 30616 external_mini_cluster.cc:1428] Setting key ce719b2e4b5b45f12adc535f8ab42777
I20260810 15:19:50.872970 673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: Bootstrap starting.
I20260810 15:19:50.873700 673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:50.874073 673 log.cc:824] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:50.874816 673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: No bootstrap required, opened a new log
I20260810 15:19:50.876936 673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } }
I20260810 15:19:50.877068 673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:50.877092 673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a12c691fa5b48beba7eae13d47613b7, State: Initialized, Role: FOLLOWER
I20260810 15:19:50.877216 673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } }
I20260810 15:19:50.877298 673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:50.877333 673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:50.877379 673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:50.877934 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:50.872508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.29.230.62:33905)
0810 15:19:50.872730 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:50.872743 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:50.872775 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:19:50.873290 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:50.873301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:50.873386 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:50.873547 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:50.873756 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.875002 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.875807 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:50.876403 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:50.876549 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:50.877151 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:50.877154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:50.877155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:50.877351 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:50.877393 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:50.877408 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:50.877479 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:50.877551 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 15:19:50.877734 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260810 15:19:50.878046 673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } }
I20260810 15:19:50.878140 673 leader_election.cc:302] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a12c691fa5b48beba7eae13d47613b7; no voters:
I20260810 15:19:50.878261 673 leader_election.cc:288] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:50.878309 681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:50.878410 681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [term 1 LEADER]: Becoming Leader. State: Replica: 6a12c691fa5b48beba7eae13d47613b7, State: Running, Role: LEADER
I20260810 15:19:50.878507 673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:50.878489 681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } }
I20260810 15:19:50.878866 681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a12c691fa5b48beba7eae13d47613b7. Latest consensus state: current_term: 1 leader_uuid: "6a12c691fa5b48beba7eae13d47613b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } } }
I20260810 15:19:50.878939 681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:50.879092 683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a12c691fa5b48beba7eae13d47613b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a12c691fa5b48beba7eae13d47613b7" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33905 } } }
I20260810 15:19:50.879148 683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:50.879280 694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:50.879765 694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:50.881177 694 catalog_manager.cc:1347] Generated new cluster ID: b7692a3059ed43aab9b5f3fe45f87c4f
I20260810 15:19:50.881224 694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:50.888001 694 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:50.888511 694 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:50.890970 694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: Generated new TSK 0
I20260810 15:19:50.891136 694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:50.893239 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.1:0
--local_ip_for_outbound_sockets=127.29.230.1
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--builtin_ntp_servers=127.29.230.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:50.984176 720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:50.984375 720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:50.984395 720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:50.985514 720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:50.985554 720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:50.985579 720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.1
I20260810 15:19:50.986959 720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.230.1
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.1
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:50.987197 720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:50.987524 720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:50.993400 731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.993425 727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:50.993582 726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:50.993649 720 server_base.cc:1034] running on GCE node
I20260810 15:19:50.993916 720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:50.994167 720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:50.995344 720 hybrid_clock.cc:648] HybridClock initialized: now 1786375190995323 us; error 33 us; skew 500 ppm
I20260810 15:19:50.996625 720 webserver.cc:460] Webserver started at http://127.29.230.1:37293/ using document root <none> and password file <none>
I20260810 15:19:50.996883 720 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:50.996928 720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:50.997071 720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:50.998169 720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data/instance:
uuid: "5eafdd70f4b8445d9280ffe99d7c19d0"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "ed66760c112abca693210e57749e5db5"
server_key_iv: "8926321758b875d70025ce3f16689a42"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.998499 720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance:
uuid: "5eafdd70f4b8445d9280ffe99d7c19d0"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "ed66760c112abca693210e57749e5db5"
server_key_iv: "8926321758b875d70025ce3f16689a42"
server_key_version: "encryptionkey@0"
I20260810 15:19:50.999821 720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:51.000550 742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.000699 720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:51.000778 720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal
uuid: "5eafdd70f4b8445d9280ffe99d7c19d0"
format_stamp: "Formatted at 2026-08-10 15:19:50 on dist-test-slave-gjrz"
server_key: "ed66760c112abca693210e57749e5db5"
server_key_iv: "8926321758b875d70025ce3f16689a42"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.000849 720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:51.016026 720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:51.016323 720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:51.016453 720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:51.016680 720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:51.017005 720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:51.017041 720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.017071 720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:51.017097 720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.022765 720 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.1:40069
I20260810 15:19:51.022853 940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.1:40069 every 8 connection(s)
I20260810 15:19:51.023147 720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/data/info.pb
I20260810 15:19:51.028314 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.023867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.1:35503 (local address 127.29.230.62:33905)
0810 15:19:51.023924 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.023926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.024154 (+ 228us) server_negotiation.cc:407] Connection header received
0810 15:19:51.024677 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.024678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.024697 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.024750 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:51.025467 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.025977 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.026999 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.027201 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.027282 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.028050 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:51.028051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:51.028051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:51.028056 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:51.028072 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:51.028077 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:51.028121 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.028203 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.028232 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:19:51.028539 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.023840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33905 (local address 127.29.230.1:35503)
0810 15:19:51.024021 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.024057 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.024568 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.024833 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.024837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.024847 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:51.025365 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.025371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.026072 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.026073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.026900 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.026902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.027295 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.027296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.027431 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.027911 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:51.027912 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:51.027941 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:51.027963 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:51.028127 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:51.028133 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:51.028136 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.028200 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.028320 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260810 15:19:51.028997 941 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33905
I20260810 15:19:51.029078 941 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:51.029253 941 heartbeater.cc:511] Master 127.29.230.62:33905 requested a full tablet report, sending...
I20260810 15:19:51.029742 607 ts_manager.cc:194] Registered new tserver with Master: 5eafdd70f4b8445d9280ffe99d7c19d0 (127.29.230.1:40069)
I20260810 15:19:51.030349 607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.1:35503
I20260810 15:19:51.033007 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 720
I20260810 15:19:51.033106 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-0/wal/instance
I20260810 15:19:51.033278 30616 external_mini_cluster.cc:1428] Setting key c74c5c263b00968cb90b247d5eb4779f
I20260810 15:19:51.034080 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.2:0
--local_ip_for_outbound_sockets=127.29.230.2
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--builtin_ntp_servers=127.29.230.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 15:19:51.115803 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:51.116014 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:51.116035 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:51.117136 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:51.117168 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:51.117192 945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.2
I20260810 15:19:51.118451 945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.230.2
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.2
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:51.118626 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:51.118868 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:51.123791 956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:51.123837 945 server_base.cc:1034] running on GCE node
W20260810 15:19:51.123783 962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:51.123783 954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:51.124225 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:51.124454 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:51.125607 945 hybrid_clock.cc:648] HybridClock initialized: now 1786375191125591 us; error 26 us; skew 500 ppm
I20260810 15:19:51.126724 945 webserver.cc:460] Webserver started at http://127.29.230.2:36381/ using document root <none> and password file <none>
I20260810 15:19:51.126968 945 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:51.127019 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:51.127146 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:51.128248 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data/instance:
uuid: "ac30f79707c4442590652e656b5f6fc3"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "179e7c659770dc40c55797d26e4edd0d"
server_key_iv: "01c29a1990e6e35324951ba0d8eb0fbe"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.128579 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance:
uuid: "ac30f79707c4442590652e656b5f6fc3"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "179e7c659770dc40c55797d26e4edd0d"
server_key_iv: "01c29a1990e6e35324951ba0d8eb0fbe"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.129803 945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:51.130709 969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.130894 945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:51.130996 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal
uuid: "ac30f79707c4442590652e656b5f6fc3"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "179e7c659770dc40c55797d26e4edd0d"
server_key_iv: "01c29a1990e6e35324951ba0d8eb0fbe"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.131064 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:51.140089 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:51.140391 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:51.140527 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:51.140748 945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:51.141084 945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:51.141120 945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.141142 945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:51.141166 945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.146948 945 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.2:42739
I20260810 15:19:51.147047 1174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.2:42739 every 8 connection(s)
I20260810 15:19:51.147403 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/data/info.pb
I20260810 15:19:51.149420 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 945
I20260810 15:19:51.149523 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal/instance
I20260810 15:19:51.149712 30616 external_mini_cluster.cc:1428] Setting key 3db4564fbd5af66aef7dbdf84464f727
I20260810 15:19:51.150761 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.230.3:0
--local_ip_for_outbound_sockets=127.29.230.3
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--builtin_ntp_servers=127.29.230.20:44227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 15:19:51.153105 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.148111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:56887 (local address 127.29.230.62:33905)
0810 15:19:51.148154 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.148156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.148390 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:19:51.148884 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.148886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.148897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.148952 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:51.149643 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.150312 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.151674 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.151871 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.151927 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.152839 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:51.152840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:51.152840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:51.152845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:51.152862 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:51.152866 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:51.152912 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.152997 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.153026 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:19:51.153352 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.148093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33905 (local address 127.29.230.2:56887)
0810 15:19:51.148278 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.148313 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.148796 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.149006 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.149009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.149017 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:51.149512 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.149516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.150406 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.150407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.151592 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.151595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.151959 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.151960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.152115 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.152697 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:51.152697 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:51.152729 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:51.152755 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:51.152918 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:51.152924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:51.152927 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.152995 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.153103 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260810 15:19:51.153782 1179 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33905
I20260810 15:19:51.153867 1179 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:51.154049 1179 heartbeater.cc:511] Master 127.29.230.62:33905 requested a full tablet report, sending...
I20260810 15:19:51.154392 607 ts_manager.cc:194] Registered new tserver with Master: ac30f79707c4442590652e656b5f6fc3 (127.29.230.2:42739)
I20260810 15:19:51.154749 607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.2:56887
W20260810 15:19:51.232105 1188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:51.232312 1188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:51.232342 1188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:51.233503 1188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:51.233554 1188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:51.233587 1188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.230.3
I20260810 15:19:51.234941 1188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:44227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.230.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.230.3
--webserver_port=0
--tserver_master_addrs=127.29.230.62:33905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.230.3
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:51.235162 1188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:51.235461 1188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:51.240223 1200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:51.240200 1203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:51.240262 1201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:51.240479 1188 server_base.cc:1034] running on GCE node
I20260810 15:19:51.240638 1188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:51.240825 1188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:51.241962 1188 hybrid_clock.cc:648] HybridClock initialized: now 1786375191241945 us; error 31 us; skew 500 ppm
I20260810 15:19:51.243211 1188 webserver.cc:460] Webserver started at http://127.29.230.3:42361/ using document root <none> and password file <none>
I20260810 15:19:51.243445 1188 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:51.243499 1188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:51.243618 1188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:51.244658 1188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data/instance:
uuid: "cfb68de9a2a54742b05f64b9608f8ebc"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "99bb9cecdc47c54452698b82fc4a5127"
server_key_iv: "f307935b7a90b1d8eca89637b3983b54"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.244968 1188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance:
uuid: "cfb68de9a2a54742b05f64b9608f8ebc"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "99bb9cecdc47c54452698b82fc4a5127"
server_key_iv: "f307935b7a90b1d8eca89637b3983b54"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.246311 1188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:51.247081 1213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.247305 1188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:51.247394 1188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal
uuid: "cfb68de9a2a54742b05f64b9608f8ebc"
format_stamp: "Formatted at 2026-08-10 15:19:51 on dist-test-slave-gjrz"
server_key: "99bb9cecdc47c54452698b82fc4a5127"
server_key_iv: "f307935b7a90b1d8eca89637b3983b54"
server_key_version: "encryptionkey@0"
I20260810 15:19:51.247463 1188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:51.256075 1188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:51.256351 1188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:51.256458 1188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:51.256641 1188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 15:19:51.256942 1188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 15:19:51.256973 1188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.256994 1188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 15:19:51.257026 1188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:51.262816 1188 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.3:38583
I20260810 15:19:51.262862 1431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.3:38583 every 8 connection(s)
I20260810 15:19:51.263219 1188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/data/info.pb
I20260810 15:19:51.266546 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1188
I20260810 15:19:51.266670 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal/instance
I20260810 15:19:51.266865 30616 external_mini_cluster.cc:1428] Setting key b391b6c6f66def6e7843a1a8d6607b0d
I20260810 15:19:51.268667 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.264056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.3:44277 (local address 127.29.230.62:33905)
0810 15:19:51.264139 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.264142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.264300 (+ 158us) server_negotiation.cc:407] Connection header received
0810 15:19:51.264816 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.264817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.264829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.264885 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:51.265547 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.266040 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.267319 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.267544 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.267602 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.268455 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:51.268457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:51.268457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:51.268461 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:51.268474 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:51.268478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:51.268529 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.268586 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.268609 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 15:19:51.268868 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.264014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:33905 (local address 127.29.230.3:44277)
0810 15:19:51.264184 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.264219 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.264703 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.264944 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.264947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.264955 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:19:51.265462 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.265467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.266126 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.266127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.267193 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.267196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.267623 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.267624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.267788 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.268323 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 15:19:51.268324 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:19:51.268354 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 15:19:51.268378 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:19:51.268534 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:19:51.268536 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:19:51.268538 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.268582 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.268662 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260810 15:19:51.269290 1433 heartbeater.cc:347] Connected to a master server at 127.29.230.62:33905
I20260810 15:19:51.269371 1433 heartbeater.cc:464] Registering TS with master...
I20260810 15:19:51.269549 1433 heartbeater.cc:511] Master 127.29.230.62:33905 requested a full tablet report, sending...
I20260810 15:19:51.269869 607 ts_manager.cc:194] Registered new tserver with Master: cfb68de9a2a54742b05f64b9608f8ebc (127.29.230.3:38583)
I20260810 15:19:51.270246 607 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.230.3:44277
I20260810 15:19:51.278559 30616 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 15:19:51.282620 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.279964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.29.230.62:33905)
0810 15:19:51.280018 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.280021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.280085 (+ 64us) server_negotiation.cc:407] Connection header received
0810 15:19:51.280146 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.280147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.280157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.280203 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:51.280349 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.280817 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.281557 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.281760 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.281814 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.282409 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:51.282410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:51.282411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:51.282415 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:51.282429 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:51.282434 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:51.282478 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.282524 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.282547 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:19:51.284130 607 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 15:19:51.285663 607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 15:19:51.291667 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.287741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42739 (local address 127.0.0.1:38092)
0810 15:19:51.287901 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.287941 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.288002 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.288209 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.288212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.288214 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:51.288287 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.288289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289305 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.289306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.290415 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.290418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291366 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.291449 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.291465 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.291540 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.291596 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260810 15:19:51.291780 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.287930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:38583 (local address 127.0.0.1:51892)
0810 15:19:51.288182 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.288192 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.288244 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.288626 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.288627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.288628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:51.288656 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.288657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289935 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.289936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.290772 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.290775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291574 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.291627 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.291630 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.291673 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.291705 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":115,"threads_started":1}
I20260810 15:19:51.291682 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.287834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.29.230.2:42739)
0810 15:19:51.288059 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.288063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.288073 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:51.288113 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.288115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.288132 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.288206 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:51.288370 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289227 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.290511 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291288 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291354 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.291447 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.291564 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.291601 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260810 15:19:51.291975 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.288174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.29.230.3:38583)
0810 15:19:51.288432 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.288435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.288444 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:51.288477 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.288479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.288492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.288565 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:51.288730 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289841 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.290844 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291393 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291455 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.291540 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.291860 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.291900 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260810 15:19:51.292176 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.287742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:40069 (local address 127.0.0.1:60872)
0810 15:19:51.288027 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.288037 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.288094 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.288542 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.288544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.288546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:51.288585 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.288586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289972 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.289974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.291023 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.291026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291973 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.292037 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.292042 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.292071 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.292110 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
I20260810 15:19:51.292531 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.287834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60872 (local address 127.29.230.1:40069)
0810 15:19:51.288345 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.288349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.288358 (+ 9us) server_negotiation.cc:407] Connection header received
0810 15:19:51.288395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.288397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.288411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.288719 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:51.288780 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.289841 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.291118 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.291880 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.292285 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.292379 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.292404 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.292444 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":97,"threads_started":1}
I20260810 15:19:51.292531 1060 tablet_service.cc:1467] Processing CreateTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 (DEFAULT_TABLE table=test-table [id=7641435db2934393946c9e51f03e1750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:51.292924 1060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a217f1e50fe4d1d9203afa440c681c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:51.293229 826 tablet_service.cc:1467] Processing CreateTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 (DEFAULT_TABLE table=test-table [id=7641435db2934393946c9e51f03e1750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:51.293510 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a217f1e50fe4d1d9203afa440c681c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:51.295477 1454 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Bootstrap starting.
I20260810 15:19:51.295755 1307 tablet_service.cc:1467] Processing CreateTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 (DEFAULT_TABLE table=test-table [id=7641435db2934393946c9e51f03e1750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 15:19:51.296103 1307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a217f1e50fe4d1d9203afa440c681c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:51.296129 1454 tablet_bootstrap.cc:654] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:51.296362 1454 log.cc:824] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:51.296360 1455 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Bootstrap starting.
I20260810 15:19:51.296998 1455 tablet_bootstrap.cc:654] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:51.297019 1454 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: No bootstrap required, opened a new log
I20260810 15:19:51.297070 1454 ts_tablet_manager.cc:1397] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:51.297322 1455 log.cc:824] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:51.298143 1455 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: No bootstrap required, opened a new log
I20260810 15:19:51.298218 1455 ts_tablet_manager.cc:1397] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:51.299166 1454 raft_consensus.cc:348] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.299392 1454 raft_consensus.cc:374] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:51.299435 1454 raft_consensus.cc:729] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac30f79707c4442590652e656b5f6fc3, State: Initialized, Role: FOLLOWER
I20260810 15:19:51.299585 1454 consensus_queue.cc:260] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.299867 1454 ts_tablet_manager.cc:1428] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:51.299885 1460 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Bootstrap starting.
I20260810 15:19:51.299999 1179 heartbeater.cc:503] Master 127.29.230.62:33905 was elected leader, sending a full tablet report...
I20260810 15:19:51.300576 1460 tablet_bootstrap.cc:654] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:51.300886 1460 log.cc:824] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:51.300825 1455 raft_consensus.cc:348] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.300974 1455 raft_consensus.cc:374] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:51.301007 1455 raft_consensus.cc:729] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eafdd70f4b8445d9280ffe99d7c19d0, State: Initialized, Role: FOLLOWER
I20260810 15:19:51.301131 1455 consensus_queue.cc:260] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.301390 1455 ts_tablet_manager.cc:1428] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 15:19:51.301527 941 heartbeater.cc:503] Master 127.29.230.62:33905 was elected leader, sending a full tablet report...
I20260810 15:19:51.301733 1460 tablet_bootstrap.cc:492] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: No bootstrap required, opened a new log
I20260810 15:19:51.301797 1460 ts_tablet_manager.cc:1397] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 15:19:51.303895 1460 raft_consensus.cc:348] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.304066 1460 raft_consensus.cc:374] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:51.304092 1460 raft_consensus.cc:729] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfb68de9a2a54742b05f64b9608f8ebc, State: Initialized, Role: FOLLOWER
I20260810 15:19:51.304219 1460 consensus_queue.cc:260] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.304461 1460 ts_tablet_manager.cc:1428] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 15:19:51.304513 1433 heartbeater.cc:503] Master 127.29.230.62:33905 was elected leader, sending a full tablet report...
I20260810 15:19:51.358038 1462 raft_consensus.cc:482] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 15:19:51.358170 1462 raft_consensus.cc:504] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.358541 1462 leader_election.cc:288] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5eafdd70f4b8445d9280ffe99d7c19d0 (127.29.230.1:40069), cfb68de9a2a54742b05f64b9608f8ebc (127.29.230.3:38583)
I20260810 15:19:51.361435 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.358710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.3:38583 (local address 127.29.230.2:34535)
0810 15:19:51.358766 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.358781 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.358842 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.359006 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.359007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.359009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:51.359091 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.359093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.359807 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.359808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.360561 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.360563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.361199 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.361200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.361271 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.361274 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.361319 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.361360 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 15:19:51.361434 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.358814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:34535 (local address 127.29.230.3:38583)
0810 15:19:51.358883 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.358885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.358895 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:19:51.358931 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.358932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.358942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.358993 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:51.359181 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.359726 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.360658 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.361134 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.361199 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.361208 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.361333 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.361373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 15:19:51.361569 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.358710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:40069 (local address 127.29.230.2:55873)
0810 15:19:51.358880 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.358890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:19:51.358945 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:19:51.359111 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:19:51.359112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:19:51.359114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 15:19:51.359194 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.359196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.359838 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.359839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.360682 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:19:51.360685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.361308 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.361310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:19:51.361395 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.361399 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 15:19:51.361445 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 15:19:51.361483 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260810 15:19:51.361564 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.358858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.230.2:55873 (local address 127.29.230.1:40069)
0810 15:19:51.358911 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.358913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.358936 (+ 23us) server_negotiation.cc:407] Connection header received
0810 15:19:51.359009 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.359010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.359020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.359079 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 15:19:51.359299 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.359761 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.360759 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.361230 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.361296 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.361304 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.361459 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.361504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:19:51.361814 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a217f1e50fe4d1d9203afa440c681c9" candidate_uuid: "ac30f79707c4442590652e656b5f6fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" is_pre_election: true
I20260810 15:19:51.361953 862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a217f1e50fe4d1d9203afa440c681c9" candidate_uuid: "ac30f79707c4442590652e656b5f6fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" is_pre_election: true
I20260810 15:19:51.362010 1345 raft_consensus.cc:2454] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac30f79707c4442590652e656b5f6fc3 in term 0.
I20260810 15:19:51.362084 862 raft_consensus.cc:2454] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac30f79707c4442590652e656b5f6fc3 in term 0.
I20260810 15:19:51.362244 972 leader_election.cc:302] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac30f79707c4442590652e656b5f6fc3, cfb68de9a2a54742b05f64b9608f8ebc; no voters:
I20260810 15:19:51.362412 1462 raft_consensus.cc:2781] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 15:19:51.362488 1462 raft_consensus.cc:482] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 15:19:51.362511 1462 raft_consensus.cc:3037] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:51.363212 1462 raft_consensus.cc:504] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.363363 1462 leader_election.cc:288] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [CANDIDATE]: Term 1 election: Requested vote from peers 5eafdd70f4b8445d9280ffe99d7c19d0 (127.29.230.1:40069), cfb68de9a2a54742b05f64b9608f8ebc (127.29.230.3:38583)
I20260810 15:19:51.363523 1345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a217f1e50fe4d1d9203afa440c681c9" candidate_uuid: "ac30f79707c4442590652e656b5f6fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfb68de9a2a54742b05f64b9608f8ebc"
I20260810 15:19:51.363615 1345 raft_consensus.cc:3037] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:51.363628 862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a217f1e50fe4d1d9203afa440c681c9" candidate_uuid: "ac30f79707c4442590652e656b5f6fc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0"
I20260810 15:19:51.363693 862 raft_consensus.cc:3037] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:51.364553 1345 raft_consensus.cc:2454] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac30f79707c4442590652e656b5f6fc3 in term 1.
I20260810 15:19:51.364553 862 raft_consensus.cc:2454] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac30f79707c4442590652e656b5f6fc3 in term 1.
I20260810 15:19:51.364745 972 leader_election.cc:302] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac30f79707c4442590652e656b5f6fc3, cfb68de9a2a54742b05f64b9608f8ebc; no voters:
I20260810 15:19:51.364857 1462 raft_consensus.cc:2781] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:51.365023 1462 raft_consensus.cc:686] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 1 LEADER]: Becoming Leader. State: Replica: ac30f79707c4442590652e656b5f6fc3, State: Running, Role: LEADER
I20260810 15:19:51.365111 1462 consensus_queue.cc:237] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } }
I20260810 15:19:51.365968 606 catalog_manager.cc:5495] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac30f79707c4442590652e656b5f6fc3 (127.29.230.2). New cstate: current_term: 1 leader_uuid: "ac30f79707c4442590652e656b5f6fc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac30f79707c4442590652e656b5f6fc3" member_type: VOTER last_known_addr { host: "127.29.230.2" port: 42739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 } health_report { overall_health: UNKNOWN } } }
I20260810 15:19:51.384357 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.382031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.29.230.2:42739)
0810 15:19:51.382109 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:51.382111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:51.382139 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:19:51.382177 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:51.382178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:51.382197 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:51.382249 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:19:51.382346 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.382818 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.383667 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:51.383863 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:51.383919 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:51.384060 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:19:51.384177 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:19:51.384239 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:51.384254 (+ 15us) server_negotiation.cc:299] Negotiation successful
0810 15:19:51.384273 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 15:19:51.386217 1345 raft_consensus.cc:1261] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 1 FOLLOWER]: Refusing update from remote peer ac30f79707c4442590652e656b5f6fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:51.386214 862 raft_consensus.cc:1261] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 1 FOLLOWER]: Refusing update from remote peer ac30f79707c4442590652e656b5f6fc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:19:51.386598 1470 consensus_queue.cc:1035] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5eafdd70f4b8445d9280ffe99d7c19d0" member_type: VOTER last_known_addr { host: "127.29.230.1" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:51.386750 1462 consensus_queue.cc:1035] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfb68de9a2a54742b05f64b9608f8ebc" member_type: VOTER last_known_addr { host: "127.29.230.3" port: 38583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:19:51.389741 1474 mvcc.cc:205] Tried to move back new op lower bound from 7316992783915069440 to 7316992783832100864. Current Snapshot: MvccSnapshot[applied={T|T < 7316992783915069440}]
I20260810 15:19:51.391960 606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:51.392064 606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 15:19:51.392771 606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a12c691fa5b48beba7eae13d47613b7: Sending DeleteTablet for 3 replicas of tablet 9a217f1e50fe4d1d9203afa440c681c9
I20260810 15:19:51.393134 1060 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:51 UTC) from {username='slave'} at 127.0.0.1:38092
I20260810 15:19:51.393150 826 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:51 UTC) from {username='slave'} at 127.0.0.1:60872
I20260810 15:19:51.393381 1503 tablet_replica.cc:330] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: stopping tablet replica
I20260810 15:19:51.393461 1307 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a217f1e50fe4d1d9203afa440c681c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 15:19:51 UTC) from {username='slave'} at 127.0.0.1:51892
I20260810 15:19:51.393496 1503 raft_consensus.cc:2229] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:51.393590 1503 raft_consensus.cc:2258] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:51.393693 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 720
I20260810 15:19:51.394001 1503 ts_tablet_manager.cc:1905] T 9a217f1e50fe4d1d9203afa440c681c9 P 5eafdd70f4b8445d9280ffe99d7c19d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:51.394423 1504 tablet_replica.cc:330] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: stopping tablet replica
I20260810 15:19:51.394528 1504 raft_consensus.cc:2229] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 15:19:51.394624 1504 raft_consensus.cc:2258] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:51.397276 1504 ts_tablet_manager.cc:1905] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 15:19:51.398633 1504 ts_tablet_manager.cc:1918] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:51.398698 1504 log.cc:1196] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-2/wal/wals/9a217f1e50fe4d1d9203afa440c681c9
I20260810 15:19:51.398895 1504 ts_tablet_manager.cc:1939] T 9a217f1e50fe4d1d9203afa440c681c9 P cfb68de9a2a54742b05f64b9608f8ebc: Deleting consensus metadata
I20260810 15:19:51.399286 585 catalog_manager.cc:4841] TS cfb68de9a2a54742b05f64b9608f8ebc (127.29.230.3:38583): tablet 9a217f1e50fe4d1d9203afa440c681c9 (table test-table [id=7641435db2934393946c9e51f03e1750]) successfully deleted
I20260810 15:19:51.407413 1502 tablet_replica.cc:330] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: stopping tablet replica
I20260810 15:19:51.407543 1502 raft_consensus.cc:2229] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 15:19:51.407671 1502 raft_consensus.cc:2258] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 15:19:51.408124 1502 ts_tablet_manager.cc:1905] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 15:19:51.409358 589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.230.1:40069 (error 108)
I20260810 15:19:51.409646 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 945
I20260810 15:19:51.409794 1502 ts_tablet_manager.cc:1918] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 15:19:51.409861 1502 log.cc:1196] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Deleting WAL directory at /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786374947162751-30616-0/minicluster-data/ts-1/wal/wals/9a217f1e50fe4d1d9203afa440c681c9
I20260810 15:19:51.410032 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.409884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.1:40069 (local address 127.0.0.1:60886)
0810 15:19:51.409957 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.409996 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.1:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 15:19:51.410095 1502 ts_tablet_manager.cc:1939] T 9a217f1e50fe4d1d9203afa440c681c9 P ac30f79707c4442590652e656b5f6fc3: Deleting consensus metadata
W20260810 15:19:51.410257 589 catalog_manager.cc:4584] TS 5eafdd70f4b8445d9280ffe99d7c19d0 (127.29.230.1:40069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a217f1e50fe4d1d9203afa440c681c9: Network error: Client connection negotiation failed: client connection to 127.29.230.1:40069: connect: Connection refused (error 111)
I20260810 15:19:51.417902 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1188
I20260810 15:19:51.417969 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:51.417858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.2:42739 (local address 127.0.0.1:38108)
0810 15:19:51.417914 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:19:51.417939 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.2:42739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 15:19:51.442775 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 557
2026-08-10T15:19:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (728 ms)
[----------] 6 tests from EncryptionPolicyTest (9057 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:51 dist-test-slave-gjrz krb5kdc[1516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:51 dist-test-slave-gjrz krb5kdc[1516](info): set up 2 sockets
Aug 10 15:19:51 dist-test-slave-gjrz krb5kdc[1516](info): commencing operation
krb5kdc: starting...
W20260810 15:19:52.625420 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:19:52.746773 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:52.829524 1539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:52.829759 1539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:52.829793 1539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:52.830991 1539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:52.831039 1539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:52.831058 1539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:52.831075 1539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:52.831089 1539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:52.832683 1539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:52.832932 1539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:52.833144 1539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:52.837952 1549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:52.837971 1547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:52.837990 1539 server_base.cc:1034] running on GCE node
W20260810 15:19:52.837975 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:52.838433 1539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:52.838752 1539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:52.839888 1539 hybrid_clock.cc:648] HybridClock initialized: now 1786375192839875 us; error 27 us; skew 500 ppm
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:52.842320 1539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:19:52.842747 1539 webserver.cc:460] Webserver started at http://127.0.0.1:35943/ using document root <none> and password file <none>
I20260810 15:19:52.842940 1539 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:52.842984 1539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:52.843122 1539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:52.844237 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "971ac9b821dc4b7db970f14ffcb2d8e2"
format_stamp: "Formatted at 2026-08-10 15:19:52 on dist-test-slave-gjrz"
server_key: "9bf797b9a17ba471db91d4bd048dd5c7"
server_key_iv: "c66ee6d24ec586d7a3d6c92cfd33d26e"
server_key_version: "encryptionkey@0"
I20260810 15:19:52.844581 1539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "971ac9b821dc4b7db970f14ffcb2d8e2"
format_stamp: "Formatted at 2026-08-10 15:19:52 on dist-test-slave-gjrz"
server_key: "9bf797b9a17ba471db91d4bd048dd5c7"
server_key_iv: "c66ee6d24ec586d7a3d6c92cfd33d26e"
server_key_version: "encryptionkey@0"
I20260810 15:19:52.845920 1539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:52.846792 1564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:52.846977 1539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:52.847093 1539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "971ac9b821dc4b7db970f14ffcb2d8e2"
format_stamp: "Formatted at 2026-08-10 15:19:52 on dist-test-slave-gjrz"
server_key: "9bf797b9a17ba471db91d4bd048dd5c7"
server_key_iv: "c66ee6d24ec586d7a3d6c92cfd33d26e"
server_key_version: "encryptionkey@0"
I20260810 15:19:52.847153 1539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:52.854218 1539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:52.856249 1539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:52.856359 1539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:52.860332 1539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40411
I20260810 15:19:52.860397 1670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40411 every 8 connection(s)
I20260810 15:19:52.860693 1539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:52.861572 1671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:52.862226 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1539
I20260810 15:19:52.862331 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:52.862516 30616 external_mini_cluster.cc:1428] Setting key b1ddbd938b518e5bf1bbfe972ea7ffed
I20260810 15:19:52.864482 1671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2: Bootstrap starting.
I20260810 15:19:52.865077 1671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:52.865355 1671 log.cc:824] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:52.866011 1671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2: No bootstrap required, opened a new log
I20260810 15:19:52.867933 1671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } }
I20260810 15:19:52.868076 1671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:52.868109 1671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971ac9b821dc4b7db970f14ffcb2d8e2, State: Initialized, Role: FOLLOWER
I20260810 15:19:52.868251 1671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } }
I20260810 15:19:52.868330 1671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:52.868361 1671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:52.868410 1671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:19:52.869022 1671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } }
I20260810 15:19:52.869130 1671 leader_election.cc:302] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 971ac9b821dc4b7db970f14ffcb2d8e2; no voters:
I20260810 15:19:52.869309 1671 leader_election.cc:288] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:52.869370 1680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:52.869577 1671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:52.869601 1680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 971ac9b821dc4b7db970f14ffcb2d8e2, State: Running, Role: LEADER
I20260810 15:19:52.869725 1680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } }
I20260810 15:19:52.870612 1681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } } }
I20260810 15:19:52.870705 1681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:52.871076 1683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 971ac9b821dc4b7db970f14ffcb2d8e2. Latest consensus state: current_term: 1 leader_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "971ac9b821dc4b7db970f14ffcb2d8e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40411 } } }
I20260810 15:19:52.871201 1704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:52.871397 1683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:52.871850 1704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:52.873272 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:52.863040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38693 (local address 127.0.0.1:40411)
0810 15:19:52.863234 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:52.863242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:52.863292 (+ 50us) server_negotiation.cc:407] Connection header received
0810 15:19:52.863762 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:52.863769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:52.863816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:52.863939 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:52.864593 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:52.865599 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:52.866393 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:52.866829 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:52.866926 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:52.869505 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:52.869510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:52.869511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:52.869525 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:52.871545 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:52.871948 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:52.871951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:52.871952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:52.872050 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:52.872272 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:52.872273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:52.872274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:52.872562 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:52.872719 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:52.872930 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 15:19:52.873056 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260810 15:19:52.873617 1704 catalog_manager.cc:1347] Generated new cluster ID: 766108e5dd88434184666d665bec442f
I20260810 15:19:52.873706 1704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:52.874138 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:19:52.881294 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:52.875536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42445 (local address 127.0.0.1:40411)
0810 15:19:52.875574 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:52.875576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:52.875663 (+ 87us) server_negotiation.cc:407] Connection header received
0810 15:19:52.875750 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:52.875750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:52.875761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:52.875806 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:52.876282 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:52.876754 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:52.877441 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:52.877726 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:52.877785 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:52.879002 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:52.879003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:52.879003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:52.879011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:52.880249 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:52.880585 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:52.880586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:52.880587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:52.880633 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:52.880817 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:52.880818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:52.880819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:52.880918 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:52.881005 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:52.881154 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 15:19:52.881209 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:19:52.886981 1704 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:52.887507 1704 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:52.894434 1704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 971ac9b821dc4b7db970f14ffcb2d8e2: Generated new TSK 0
I20260810 15:19:52.894594 1704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:52.900136 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1539
W20260810 15:19:52.930488 1512 connection.cc:535] client connection to 127.0.0.1:40411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40411): Connection reset by peer (error 104)
2026-08-10T15:19:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1734](info): set up 2 sockets
Aug 10 15:19:52 dist-test-slave-gjrz krb5kdc[1734](info): commencing operation
krb5kdc: starting...
W20260810 15:19:54.113914 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:54 dist-test-slave-gjrz krb5kdc[1734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:19:54.233271 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:54.314361 1762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:54.314568 1762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:54.314589 1762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:54.315694 1762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:54.315727 1762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:54.315740 1762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:54.315752 1762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:54.315761 1762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:54.317179 1762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:54.317382 1762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:54.317548 1762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:54.322345 1776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:54.322376 1780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:54.322520 1762 server_base.cc:1034] running on GCE node
W20260810 15:19:54.322353 1775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:54.322832 1762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:54.323120 1762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:54.324283 1762 hybrid_clock.cc:648] HybridClock initialized: now 1786375194324267 us; error 32 us; skew 500 ppm
Aug 10 15:19:54 dist-test-slave-gjrz krb5kdc[1734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:54.326679 1762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:19:54.327083 1762 webserver.cc:460] Webserver started at http://127.0.0.1:34895/ using document root <none> and password file <none>
I20260810 15:19:54.327317 1762 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:54.327369 1762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:54.327490 1762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:54.328558 1762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "5397782e7bb24e239c1092606ae37e4f"
format_stamp: "Formatted at 2026-08-10 15:19:54 on dist-test-slave-gjrz"
server_key: "6d816c950fbe65d6cff7d551cab304f3"
server_key_iv: "461025e2ede85c6c4624e5aedab59752"
server_key_version: "encryptionkey@0"
I20260810 15:19:54.328900 1762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "5397782e7bb24e239c1092606ae37e4f"
format_stamp: "Formatted at 2026-08-10 15:19:54 on dist-test-slave-gjrz"
server_key: "6d816c950fbe65d6cff7d551cab304f3"
server_key_iv: "461025e2ede85c6c4624e5aedab59752"
server_key_version: "encryptionkey@0"
I20260810 15:19:54.330181 1762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:19:54.330834 1790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:54.330983 1762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:19:54.331079 1762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "5397782e7bb24e239c1092606ae37e4f"
format_stamp: "Formatted at 2026-08-10 15:19:54 on dist-test-slave-gjrz"
server_key: "6d816c950fbe65d6cff7d551cab304f3"
server_key_iv: "461025e2ede85c6c4624e5aedab59752"
server_key_version: "encryptionkey@0"
I20260810 15:19:54.331143 1762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:54.355460 1762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:54.357298 1762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:54.357409 1762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:54.361214 1762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33057
I20260810 15:19:54.361264 1879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33057 every 8 connection(s)
I20260810 15:19:54.361572 1762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:54.362401 1880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:54.364684 1880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f: Bootstrap starting.
I20260810 15:19:54.365240 1880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:54.365515 1880 log.cc:824] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:54.366236 1880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f: No bootstrap required, opened a new log
I20260810 15:19:54.367944 1880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 15:19:54.368089 1880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:54.368125 1880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5397782e7bb24e239c1092606ae37e4f, State: Initialized, Role: FOLLOWER
I20260810 15:19:54.368263 1880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 15:19:54.368275 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1762
I20260810 15:19:54.368326 1880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:54.368357 1880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:54.368376 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:54.368413 1880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:54.368568 30616 external_mini_cluster.cc:1428] Setting key 47ab46bf25944ffce5ddff7be0992ed9
I20260810 15:19:54.369072 1880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 15:19:54.369187 1880 leader_election.cc:302] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5397782e7bb24e239c1092606ae37e4f; no voters:
I20260810 15:19:54.369341 1880 leader_election.cc:288] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:54.369371 1892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:54.369549 1892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [term 1 LEADER]: Becoming Leader. State: Replica: 5397782e7bb24e239c1092606ae37e4f, State: Running, Role: LEADER
I20260810 15:19:54.369598 1880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:54.369659 1892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } }
I20260810 15:19:54.370004 1892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5397782e7bb24e239c1092606ae37e4f. Latest consensus state: current_term: 1 leader_uuid: "5397782e7bb24e239c1092606ae37e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260810 15:19:54.370078 1892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:54.370231 1896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5397782e7bb24e239c1092606ae37e4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5397782e7bb24e239c1092606ae37e4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33057 } } }
I20260810 15:19:54.370304 1896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:54.370406 1909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:54.370906 1909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:54.372664 1909 catalog_manager.cc:1347] Generated new cluster ID: cc800e5949e34d06a1dccae784bec026
I20260810 15:19:54.372716 1909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:54.374712 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:54.369003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51221 (local address 127.0.0.1:33057)
0810 15:19:54.369296 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:54.369305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:54.369337 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:19:54.369453 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:54.369460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:54.369533 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:54.369925 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:54.370024 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:54.371782 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:54.372601 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:54.373178 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:54.373280 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:54.374037 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:54.374040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:54.374040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:54.374236 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:54.374271 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:54.374285 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:54.374365 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:54.374483 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 15:19:54.374509 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260810 15:19:54.375669 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:19:54.380250 1909 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:54.380668 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:54.377004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39129 (local address 127.0.0.1:33057)
0810 15:19:54.377056 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:54.377058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:54.377269 (+ 211us) server_negotiation.cc:407] Connection header received
0810 15:19:54.377296 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:54.377297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:54.377330 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:54.377385 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:54.377510 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:54.378307 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:54.379308 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:54.379638 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:54.379710 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:54.380413 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:54.380415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:54.380415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:54.380420 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:54.380434 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:54.380438 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:54.380500 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:54.380582 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 15:19:54.380592 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 15:19:54.381050 1909 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:54.387635 1909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5397782e7bb24e239c1092606ae37e4f: Generated new TSK 0
I20260810 15:19:54.387794 1909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:54.389379 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1762
W20260810 15:19:54.403735 1728 connection.cc:535] client connection to 127.0.0.1:33057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33057): Connection reset by peer (error 104)
2026-08-10T15:19:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:54 dist-test-slave-gjrz krb5kdc[1946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:54 dist-test-slave-gjrz krb5kdc[1946](info): set up 2 sockets
Aug 10 15:19:54 dist-test-slave-gjrz krb5kdc[1946](info): commencing operation
krb5kdc: starting...
W20260810 15:19:55.575382 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:55 dist-test-slave-gjrz krb5kdc[1946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:19:55.697719 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:55.778594 1964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:55.778797 1964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:55.778828 1964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:55.779980 1964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:55.780022 1964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:55.780040 1964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:55.780057 1964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:55.780071 1964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:55.781598 1964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:55.781845 1964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:55.782063 1964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:55.786674 1974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:55.786705 1972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:55.786690 1970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:55.786876 1964 server_base.cc:1034] running on GCE node
I20260810 15:19:55.787122 1964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:55.787410 1964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:55.788573 1964 hybrid_clock.cc:648] HybridClock initialized: now 1786375195788546 us; error 41 us; skew 500 ppm
Aug 10 15:19:55 dist-test-slave-gjrz krb5kdc[1946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:55.791098 1964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:19:55.791566 1964 webserver.cc:460] Webserver started at http://127.0.0.1:36539/ using document root <none> and password file <none>
I20260810 15:19:55.791787 1964 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:55.791836 1964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:55.791960 1964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:55.793025 1964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "e6019f0a82764fcb9ac6aff2b362bb5d"
format_stamp: "Formatted at 2026-08-10 15:19:55 on dist-test-slave-gjrz"
server_key: "89195e58b8c17108636d2ab6134cbbcb"
server_key_iv: "b2209905c56badc7289bd850903040c6"
server_key_version: "encryptionkey@0"
I20260810 15:19:55.793359 1964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "e6019f0a82764fcb9ac6aff2b362bb5d"
format_stamp: "Formatted at 2026-08-10 15:19:55 on dist-test-slave-gjrz"
server_key: "89195e58b8c17108636d2ab6134cbbcb"
server_key_iv: "b2209905c56badc7289bd850903040c6"
server_key_version: "encryptionkey@0"
I20260810 15:19:55.794605 1964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:55.795241 1985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:55.795433 1964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:19:55.795535 1964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "e6019f0a82764fcb9ac6aff2b362bb5d"
format_stamp: "Formatted at 2026-08-10 15:19:55 on dist-test-slave-gjrz"
server_key: "89195e58b8c17108636d2ab6134cbbcb"
server_key_iv: "b2209905c56badc7289bd850903040c6"
server_key_version: "encryptionkey@0"
I20260810 15:19:55.795603 1964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:55.799007 1964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:55.801061 1964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:55.801189 1964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:55.805073 1964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39093
I20260810 15:19:55.805121 2074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39093 every 8 connection(s)
I20260810 15:19:55.805416 1964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:55.806217 2076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:55.808434 2076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: Bootstrap starting.
I20260810 15:19:55.809002 2076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:55.809260 2076 log.cc:824] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:55.809921 2076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: No bootstrap required, opened a new log
I20260810 15:19:55.811520 2076 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } }
I20260810 15:19:55.811664 2076 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:55.811687 2076 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6019f0a82764fcb9ac6aff2b362bb5d, State: Initialized, Role: FOLLOWER
I20260810 15:19:55.811810 2076 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } }
I20260810 15:19:55.811870 2076 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:55.811893 2076 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:55.811937 2076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:55.812428 2076 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } }
I20260810 15:19:55.812520 2076 leader_election.cc:302] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6019f0a82764fcb9ac6aff2b362bb5d; no voters:
I20260810 15:19:55.812712 2076 leader_election.cc:288] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:55.812742 2083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:55.812813 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 1964
I20260810 15:19:55.812916 2083 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [term 1 LEADER]: Becoming Leader. State: Replica: e6019f0a82764fcb9ac6aff2b362bb5d, State: Running, Role: LEADER
I20260810 15:19:55.812945 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:55.812980 2076 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:55.813091 2083 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } }
I20260810 15:19:55.813151 30616 external_mini_cluster.cc:1428] Setting key a333747292eb5b224947009c396691e1
I20260810 15:19:55.814049 2085 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6019f0a82764fcb9ac6aff2b362bb5d. Latest consensus state: current_term: 1 leader_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } } }
I20260810 15:19:55.814143 2085 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:55.814229 2086 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6019f0a82764fcb9ac6aff2b362bb5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39093 } } }
I20260810 15:19:55.814278 2086 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d [sys.catalog]: This master's current role is: LEADER
W20260810 15:19:55.814900 2121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:19:55.814954 2121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:19:55.815017 2116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:55.815186 2116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:55.815528 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:55.813799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51851 (local address 127.0.0.1:39093)
0810 15:19:55.814045 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:55.814055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:55.814093 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:19:55.814236 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:55.814242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:55.814275 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:55.814409 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:55.815082 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:55.815086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:55.815086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:55.815091 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:55.815135 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:55.815149 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:55.815225 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:55.815314 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 15:19:55.815353 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20260810 15:19:55.816569 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:19:55.816660 2116 catalog_manager.cc:1347] Generated new cluster ID: fab93540c7e04dc7b21d6bfa5036fd24
I20260810 15:19:55.816701 2116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:55.820317 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:55.818164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46909 (local address 127.0.0.1:39093)
0810 15:19:55.818197 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:55.818198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:55.819440 (+ 1242us) server_negotiation.cc:407] Connection header received
0810 15:19:55.819547 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:55.819548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:55.819554 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:55.819609 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:55.820130 (+ 521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:55.820131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:55.820131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:55.820135 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:55.820149 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:55.820153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:55.820194 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:55.820248 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 15:19:55.820259 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:19:55.826072 2116 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:55.826215 2116 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:55.829890 2116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6019f0a82764fcb9ac6aff2b362bb5d: Generated new TSK 0
I20260810 15:19:55.830044 2116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:55.832989 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 1964
2026-08-10T15:19:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1447 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:55 dist-test-slave-gjrz krb5kdc[2146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:55 dist-test-slave-gjrz krb5kdc[2146](info): set up 2 sockets
Aug 10 15:19:55 dist-test-slave-gjrz krb5kdc[2146](info): commencing operation
krb5kdc: starting...
W20260810 15:19:57.029022 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:19:57.149628 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:57.235177 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:57.235406 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:57.235436 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:57.236572 2172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:57.236613 2172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:57.236625 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:57.236636 2172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:57.236646 2172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:57.238126 2172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:57.238363 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:57.238580 2172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:57.243284 2183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:57.243316 2185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:57.243383 2172 server_base.cc:1034] running on GCE node
W20260810 15:19:57.243292 2182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:57.243693 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:57.243989 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:57.245155 2172 hybrid_clock.cc:648] HybridClock initialized: now 1786375197245138 us; error 31 us; skew 500 ppm
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:57.247575 2172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:19:57.248044 2172 webserver.cc:460] Webserver started at http://127.0.0.1:40619/ using document root <none> and password file <none>
I20260810 15:19:57.248245 2172 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:57.248291 2172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:57.248409 2172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:57.249456 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb"
format_stamp: "Formatted at 2026-08-10 15:19:57 on dist-test-slave-gjrz"
server_key: "d5b7e6f8dde283412eeff275d69ba79e"
server_key_iv: "73094ae511e11c20391dd9192753bfc8"
server_key_version: "encryptionkey@0"
I20260810 15:19:57.249785 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb"
format_stamp: "Formatted at 2026-08-10 15:19:57 on dist-test-slave-gjrz"
server_key: "d5b7e6f8dde283412eeff275d69ba79e"
server_key_iv: "73094ae511e11c20391dd9192753bfc8"
server_key_version: "encryptionkey@0"
I20260810 15:19:57.251036 2172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:19:57.251773 2193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:57.251926 2172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:57.252010 2172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb"
format_stamp: "Formatted at 2026-08-10 15:19:57 on dist-test-slave-gjrz"
server_key: "d5b7e6f8dde283412eeff275d69ba79e"
server_key_iv: "73094ae511e11c20391dd9192753bfc8"
server_key_version: "encryptionkey@0"
I20260810 15:19:57.252077 2172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:57.268589 2172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:57.270406 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:57.270539 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:57.274509 2172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36597
I20260810 15:19:57.274549 2273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36597 every 8 connection(s)
I20260810 15:19:57.274880 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:57.275759 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:57.278087 2275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb: Bootstrap starting.
I20260810 15:19:57.278630 2275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:57.278863 2275 log.cc:824] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:57.279582 2275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb: No bootstrap required, opened a new log
I20260810 15:19:57.281147 2275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } }
I20260810 15:19:57.281287 2275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:57.281309 2275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54b0ff5b8b324bfd8401c73aeeb38ecb, State: Initialized, Role: FOLLOWER
I20260810 15:19:57.281409 2275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } }
I20260810 15:19:57.281461 2275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:57.281481 2275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:57.281524 2275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:57.282053 2275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } }
I20260810 15:19:57.282146 2275 leader_election.cc:302] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54b0ff5b8b324bfd8401c73aeeb38ecb; no voters:
I20260810 15:19:57.282305 2275 leader_election.cc:288] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:57.282371 2281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:57.282537 2281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [term 1 LEADER]: Becoming Leader. State: Replica: 54b0ff5b8b324bfd8401c73aeeb38ecb, State: Running, Role: LEADER
I20260810 15:19:57.282588 2275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:57.282706 2281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } }
I20260810 15:19:57.283080 2282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } } }
I20260810 15:19:57.283219 2282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:57.283227 2285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54b0ff5b8b324bfd8401c73aeeb38ecb. Latest consensus state: current_term: 1 leader_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54b0ff5b8b324bfd8401c73aeeb38ecb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36597 } } }
I20260810 15:19:57.283355 2285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:57.283695 2306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:57.284224 2306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:57.284760 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2172
I20260810 15:19:57.284883 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:57.285084 30616 external_mini_cluster.cc:1428] Setting key ff9dccd2f7c8a96b04c5d85ffcb18db4
I20260810 15:19:57.285938 2306 catalog_manager.cc:1347] Generated new cluster ID: 69e4fcacbabc44458b01526191a36d68
I20260810 15:19:57.285988 2306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:19:57.296778 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:57.285573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37115 (local address 127.0.0.1:36597)
0810 15:19:57.285840 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:57.285849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:57.285879 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:19:57.286004 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:57.286013 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:57.286078 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:57.286219 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:57.287017 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:57.288447 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:57.289549 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:57.290165 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:57.290255 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:57.293173 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:57.293177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:57.293177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:57.293188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:57.295073 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:57.295490 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:57.295492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:57.295493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:57.295592 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:57.295780 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:57.295781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:57.295781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:57.296074 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:57.296200 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:57.296323 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:19:57.296548 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":75,"threads_started":1}
I20260810 15:19:57.297699 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:19:57.305936 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:57.299143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34451 (local address 127.0.0.1:36597)
0810 15:19:57.299183 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:57.299184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:57.299191 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:19:57.299292 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:57.299293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:57.299303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:57.299365 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:57.299909 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:57.300752 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:57.301701 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:57.302031 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:57.302100 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:57.303322 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:57.303324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:57.303324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:19:57.303331 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:57.304899 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:57.305244 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:57.305245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:57.305245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:57.305304 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:19:57.305460 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:19:57.305460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:19:57.305461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:19:57.305589 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:57.305693 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:57.305769 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 15:19:57.305847 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:19:57.345317 2306 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:57.345834 2306 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:57.355937 2306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54b0ff5b8b324bfd8401c73aeeb38ecb: Generated new TSK 0
I20260810 15:19:57.356096 2306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:57.366716 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2172
2026-08-10T15:19:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2345](info): set up 2 sockets
Aug 10 15:19:57 dist-test-slave-gjrz krb5kdc[2345](info): commencing operation
krb5kdc: starting...
W20260810 15:19:58.552205 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:19:58 dist-test-slave-gjrz krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:19:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:19:58.670011 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:19:58.751554 2365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:19:58.751775 2365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:19:58.751799 2365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:19:58.753041 2365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:19:58.753089 2365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:19:58.753103 2365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:19:58.753114 2365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:19:58.753124 2365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:19:58.754642 2365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:19:58.754873 2365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:19:58.755102 2365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:19:58.759805 2380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:58.759830 2386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:19:58.759859 2382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:19:58.759905 2365 server_base.cc:1034] running on GCE node
I20260810 15:19:58.760248 2365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:19:58.760532 2365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:19:58.761685 2365 hybrid_clock.cc:648] HybridClock initialized: now 1786375198761677 us; error 35 us; skew 500 ppm
Aug 10 15:19:58 dist-test-slave-gjrz krb5kdc[2345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:19:58.767802 2365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:19:58.768304 2365 webserver.cc:460] Webserver started at http://127.0.0.1:44137/ using document root <none> and password file <none>
I20260810 15:19:58.768537 2365 fs_manager.cc:359] Metadata directory not provided
I20260810 15:19:58.768586 2365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:19:58.768716 2365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:19:58.769779 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "6957ce25d2fd4bbaa55610b2cd1e8574"
format_stamp: "Formatted at 2026-08-10 15:19:58 on dist-test-slave-gjrz"
server_key: "d68928ea7cb63d0a5cf8001936451109"
server_key_iv: "1cbabbf09034672e060499121c903d0d"
server_key_version: "encryptionkey@0"
I20260810 15:19:58.770120 2365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "6957ce25d2fd4bbaa55610b2cd1e8574"
format_stamp: "Formatted at 2026-08-10 15:19:58 on dist-test-slave-gjrz"
server_key: "d68928ea7cb63d0a5cf8001936451109"
server_key_iv: "1cbabbf09034672e060499121c903d0d"
server_key_version: "encryptionkey@0"
I20260810 15:19:58.771427 2365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:19:58.772229 2395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:19:58.772418 2365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:19:58.772498 2365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "6957ce25d2fd4bbaa55610b2cd1e8574"
format_stamp: "Formatted at 2026-08-10 15:19:58 on dist-test-slave-gjrz"
server_key: "d68928ea7cb63d0a5cf8001936451109"
server_key_iv: "1cbabbf09034672e060499121c903d0d"
server_key_version: "encryptionkey@0"
I20260810 15:19:58.772570 2365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:19:58.793152 2365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:19:58.795419 2365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:19:58.795560 2365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:19:58.799578 2365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45457
I20260810 15:19:58.799618 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45457 every 8 connection(s)
I20260810 15:19:58.799949 2365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:19:58.800794 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:19:58.803182 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574: Bootstrap starting.
I20260810 15:19:58.803812 2493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574: Neither blocks nor log segments found. Creating new log.
I20260810 15:19:58.804055 2493 log.cc:824] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574: Log is configured to *not* fsync() on all Append() calls
I20260810 15:19:58.804704 2493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574: No bootstrap required, opened a new log
I20260810 15:19:58.805490 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2365
I20260810 15:19:58.805598 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:19:58.805792 30616 external_mini_cluster.cc:1428] Setting key fca302c0569c172076d22a331c6f3b23
I20260810 15:19:58.806630 2493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260810 15:19:58.806773 2493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:19:58.806805 2493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6957ce25d2fd4bbaa55610b2cd1e8574, State: Initialized, Role: FOLLOWER
I20260810 15:19:58.806954 2493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260810 15:19:58.807045 2493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:19:58.807075 2493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:19:58.807127 2493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:19:58.807847 2493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260810 15:19:58.807956 2493 leader_election.cc:302] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6957ce25d2fd4bbaa55610b2cd1e8574; no voters:
I20260810 15:19:58.808101 2493 leader_election.cc:288] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:19:58.808140 2498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:19:58.808295 2498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [term 1 LEADER]: Becoming Leader. State: Replica: 6957ce25d2fd4bbaa55610b2cd1e8574, State: Running, Role: LEADER
I20260810 15:19:58.808344 2493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:19:58.808421 2498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } }
I20260810 15:19:58.808796 2506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6957ce25d2fd4bbaa55610b2cd1e8574. Latest consensus state: current_term: 1 leader_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260810 15:19:58.808792 2505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6957ce25d2fd4bbaa55610b2cd1e8574" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45457 } } }
I20260810 15:19:58.808892 2506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:58.808894 2505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574 [sys.catalog]: This master's current role is: LEADER
I20260810 15:19:58.809079 2515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:19:58.809620 2515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:19:58.811297 2515 catalog_manager.cc:1347] Generated new cluster ID: 5421a576366e421598e850ce975bf06e
I20260810 15:19:58.811344 2515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:19:58.812402 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:58.806323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48825 (local address 127.0.0.1:45457)
0810 15:19:58.806510 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:58.806517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:58.806547 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:19:58.806646 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:58.806651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:58.806692 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:58.806815 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:58.807177 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:58.808884 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:58.810175 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:58.810769 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:58.810865 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:58.811723 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:58.811727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:58.811727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:58.811920 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:58.811957 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:58.811971 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:58.812035 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:58.812140 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 15:19:58.812204 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260810 15:19:58.813249 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:19:58.817931 2497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:19:58.814696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50735 (local address 127.0.0.1:45457)
0810 15:19:58.814756 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 15:19:58.814757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:19:58.814854 (+ 97us) server_negotiation.cc:407] Connection header received
0810 15:19:58.814897 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:19:58.814898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:19:58.814908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:19:58.814976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:19:58.815083 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:58.815799 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:58.816748 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:19:58.817050 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:19:58.817127 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:19:58.817710 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:19:58.817711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:19:58.817712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:19:58.817716 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:19:58.817730 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:19:58.817735 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:19:58.817790 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:19:58.817836 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 15:19:58.817859 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:19:58.834491 2515 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:19:58.835064 2515 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:19:58.839833 2515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6957ce25d2fd4bbaa55610b2cd1e8574: Generated new TSK 0
I20260810 15:19:58.840013 2515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:19:58.841903 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2365
2026-08-10T15:19:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1476 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:19:58 dist-test-slave-gjrz krb5kdc[2552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:19:58 dist-test-slave-gjrz krb5kdc[2552](info): set up 2 sockets
Aug 10 15:19:58 dist-test-slave-gjrz krb5kdc[2552](info): commencing operation
krb5kdc: starting...
W20260810 15:20:00.016985 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:00 dist-test-slave-gjrz krb5kdc[2552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:00.137925 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:00.219347 2575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:00.219545 2575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:00.219575 2575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:00.220695 2575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:00.220736 2575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:00.220755 2575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:00.220772 2575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:00.220786 2575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:00.222306 2575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:00.222548 2575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:00.222760 2575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:00.227488 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:00.227499 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:00.227519 2575 server_base.cc:1034] running on GCE node
W20260810 15:20:00.227496 2586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:00.227898 2575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:00.228189 2575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:00.229323 2575 hybrid_clock.cc:648] HybridClock initialized: now 1786375200229310 us; error 29 us; skew 500 ppm
Aug 10 15:20:00 dist-test-slave-gjrz krb5kdc[2552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:00.231792 2575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:00.232251 2575 webserver.cc:460] Webserver started at http://127.0.0.1:46725/ using document root <none> and password file <none>
I20260810 15:20:00.232481 2575 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:00.232538 2575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:00.232666 2575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:00.233708 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "cf188bd4d7a7414fa5836a35079a933d"
format_stamp: "Formatted at 2026-08-10 15:20:00 on dist-test-slave-gjrz"
server_key: "c2b313c399dcafb64abd06ac8e2c21f0"
server_key_iv: "c10b4d69b1bab533c440da8cc7362a67"
server_key_version: "encryptionkey@0"
I20260810 15:20:00.234040 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "cf188bd4d7a7414fa5836a35079a933d"
format_stamp: "Formatted at 2026-08-10 15:20:00 on dist-test-slave-gjrz"
server_key: "c2b313c399dcafb64abd06ac8e2c21f0"
server_key_iv: "c10b4d69b1bab533c440da8cc7362a67"
server_key_version: "encryptionkey@0"
I20260810 15:20:00.235378 2575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:00.236095 2599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:00.236251 2575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:00.236335 2575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "cf188bd4d7a7414fa5836a35079a933d"
format_stamp: "Formatted at 2026-08-10 15:20:00 on dist-test-slave-gjrz"
server_key: "c2b313c399dcafb64abd06ac8e2c21f0"
server_key_iv: "c10b4d69b1bab533c440da8cc7362a67"
server_key_version: "encryptionkey@0"
I20260810 15:20:00.236405 2575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:00.239650 2575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:00.241317 2575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:00.241442 2575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:00.245461 2575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37805
I20260810 15:20:00.245493 2693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37805 every 8 connection(s)
I20260810 15:20:00.245795 2575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:00.246654 2694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:00.248997 2694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d: Bootstrap starting.
I20260810 15:20:00.249562 2694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:00.249837 2694 log.cc:824] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:00.250491 2694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d: No bootstrap required, opened a new log
I20260810 15:20:00.252131 2694 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } }
I20260810 15:20:00.252269 2694 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:00.252290 2694 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf188bd4d7a7414fa5836a35079a933d, State: Initialized, Role: FOLLOWER
I20260810 15:20:00.252424 2694 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } }
I20260810 15:20:00.252492 2694 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:00.252516 2694 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:00.252560 2694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:00.252991 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2575
I20260810 15:20:00.253110 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:00.253170 2694 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } }
I20260810 15:20:00.253283 2694 leader_election.cc:302] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf188bd4d7a7414fa5836a35079a933d; no voters:
I20260810 15:20:00.253304 30616 external_mini_cluster.cc:1428] Setting key e89939e9b3f6859c60972c86a4060bda
I20260810 15:20:00.253454 2694 leader_election.cc:288] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:00.253491 2699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:00.253672 2699 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [term 1 LEADER]: Becoming Leader. State: Replica: cf188bd4d7a7414fa5836a35079a933d, State: Running, Role: LEADER
I20260810 15:20:00.253718 2694 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:00.253803 2699 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } }
I20260810 15:20:00.254149 2699 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf188bd4d7a7414fa5836a35079a933d. Latest consensus state: current_term: 1 leader_uuid: "cf188bd4d7a7414fa5836a35079a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } } }
I20260810 15:20:00.254228 2699 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:00.254386 2701 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf188bd4d7a7414fa5836a35079a933d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf188bd4d7a7414fa5836a35079a933d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37805 } } }
I20260810 15:20:00.254451 2701 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:00.254549 2717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:00.255096 2717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:00.256609 2717 catalog_manager.cc:1347] Generated new cluster ID: 6b9c2ad6f03e4fcbab33f75efd51d084
I20260810 15:20:00.256659 2717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:00.256784 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:00.254279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45811 (local address 127.0.0.1:37805)
0810 15:20:00.255101 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:00.255109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:00.255137 (+ 28us) server_negotiation.cc:407] Connection header received
0810 15:20:00.255257 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:00.255285 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:00.255319 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:00.255504 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:00.256174 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:00.256178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:00.256178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:00.256182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:00.256227 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:00.256241 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:00.256505 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:00.256546 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 15:20:00.256587 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":76,"threads_started":1}
I20260810 15:20:00.257659 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:00.260172 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:00.259278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56929 (local address 127.0.0.1:37805)
0810 15:20:00.259306 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:00.259308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:00.259315 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:00.259381 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:00.259382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:00.259387 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:00.259432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:00.259987 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:00.259988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:00.259988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:00.259992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:00.260005 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:00.260009 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:00.260052 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:00.260095 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 15:20:00.260105 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 15:20:00.284240 2717 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:00.284449 2717 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:00.294294 2717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf188bd4d7a7414fa5836a35079a933d: Generated new TSK 0
I20260810 15:20:00.294509 2717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:00.305855 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2575
2026-08-10T15:20:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:00 dist-test-slave-gjrz krb5kdc[2748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:00 dist-test-slave-gjrz krb5kdc[2748](info): set up 2 sockets
Aug 10 15:20:00 dist-test-slave-gjrz krb5kdc[2748](info): commencing operation
krb5kdc: starting...
W20260810 15:20:01.496871 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:01.630313 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:01.714793 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:01.715010 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:01.715034 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:01.716279 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:01.716325 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:01.716339 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:01.716351 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:01.716361 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:01.717875 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:01.718091 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:01.718281 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:01.723100 2770 server_base.cc:1034] running on GCE node
W20260810 15:20:01.723088 2784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:01.723069 2783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:01.723074 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:01.723536 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:01.723824 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:01.724973 2770 hybrid_clock.cc:648] HybridClock initialized: now 1786375201724954 us; error 30 us; skew 500 ppm
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:01.727496 2770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:01.727967 2770 webserver.cc:460] Webserver started at http://127.0.0.1:44349/ using document root <none> and password file <none>
I20260810 15:20:01.728200 2770 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:01.728251 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:01.728381 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:01.729494 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "3a275ae65e1b4adfab1156102a741d88"
format_stamp: "Formatted at 2026-08-10 15:20:01 on dist-test-slave-gjrz"
server_key: "7a372dd44e341b70e9d00ef52e429fb4"
server_key_iv: "32e3761b467fdfc2154e812825710246"
server_key_version: "encryptionkey@0"
I20260810 15:20:01.729838 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "3a275ae65e1b4adfab1156102a741d88"
format_stamp: "Formatted at 2026-08-10 15:20:01 on dist-test-slave-gjrz"
server_key: "7a372dd44e341b70e9d00ef52e429fb4"
server_key_iv: "32e3761b467fdfc2154e812825710246"
server_key_version: "encryptionkey@0"
I20260810 15:20:01.731200 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:20:01.731956 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:01.732180 2770 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:01.732280 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "3a275ae65e1b4adfab1156102a741d88"
format_stamp: "Formatted at 2026-08-10 15:20:01 on dist-test-slave-gjrz"
server_key: "7a372dd44e341b70e9d00ef52e429fb4"
server_key_iv: "32e3761b467fdfc2154e812825710246"
server_key_version: "encryptionkey@0"
I20260810 15:20:01.732349 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:01.759914 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:01.761999 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:01.762189 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:01.766361 2770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33663
I20260810 15:20:01.766433 2889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33663 every 8 connection(s)
I20260810 15:20:01.766779 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:01.767726 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:01.770099 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: Bootstrap starting.
I20260810 15:20:01.770713 2891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:01.770988 2891 log.cc:824] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:01.771756 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: No bootstrap required, opened a new log
I20260810 15:20:01.773397 2891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260810 15:20:01.773586 2891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:01.773612 2891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a275ae65e1b4adfab1156102a741d88, State: Initialized, Role: FOLLOWER
I20260810 15:20:01.773869 2891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260810 15:20:01.773972 2891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:01.774014 2891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:01.774075 2891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:01.774649 2891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260810 15:20:01.774744 2891 leader_election.cc:302] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a275ae65e1b4adfab1156102a741d88; no voters:
I20260810 15:20:01.774914 2891 leader_election.cc:288] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:01.774952 2895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:01.775123 2895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [term 1 LEADER]: Becoming Leader. State: Replica: 3a275ae65e1b4adfab1156102a741d88, State: Running, Role: LEADER
I20260810 15:20:01.775177 2891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:01.775240 2895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } }
I20260810 15:20:01.775806 2897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a275ae65e1b4adfab1156102a741d88. Latest consensus state: current_term: 1 leader_uuid: "3a275ae65e1b4adfab1156102a741d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } } }
I20260810 15:20:01.775892 2897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:01.776125 2896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a275ae65e1b4adfab1156102a741d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a275ae65e1b4adfab1156102a741d88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33663 } } }
I20260810 15:20:01.776247 2896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:01.776243 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2770
I20260810 15:20:01.776350 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:01.776522 30616 external_mini_cluster.cc:1428] Setting key 501d07fe641e315ac3fa24df0468b59e
W20260810 15:20:01.777007 2925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:01.777066 2925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:01.777184 2914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:01.777356 2914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:01.778821 2914 catalog_manager.cc:1347] Generated new cluster ID: 6a941fecb0f34472a8cd588fecb47d16
I20260810 15:20:01.778875 2914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:01.782693 2914 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:01.783183 2914 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:20:01.787307 2914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a275ae65e1b4adfab1156102a741d88: Generated new TSK 0
I20260810 15:20:01.787470 2914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:01.788450 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:01.777050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:54255 (local address 127.0.0.1:33663)
0810 15:20:01.777214 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:01.777225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:01.777263 (+ 38us) server_negotiation.cc:407] Connection header received
0810 15:20:01.777390 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:01.777396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:01.777450 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:01.777588 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:01.778196 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:01.779606 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:01.780700 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:01.781326 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:01.781427 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:01.784427 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:01.784432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:01.784432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:01.784444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:01.786367 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:01.786844 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:01.786847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:01.786847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:01.787096 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:01.787467 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:01.787469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:01.787469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:01.787793 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:01.787941 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:01.788090 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 15:20:01.788199 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260810 15:20:01.789414 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:01.796496 2930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:01.790697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:38035 (local address 127.0.0.1:33663)
0810 15:20:01.790734 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:01.790736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:01.790839 (+ 103us) server_negotiation.cc:407] Connection header received
0810 15:20:01.790899 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:01.790900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:01.790909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:01.790950 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:01.791518 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:01.792021 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:01.792741 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:01.792944 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:01.792998 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:01.794197 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:01.794198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:01.794199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:01.794205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:01.795433 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:01.795770 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:01.795771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:01.795771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:01.795824 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:01.796025 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:01.796025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:01.796026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:01.796134 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:01.796248 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:01.796353 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 15:20:01.796415 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:20:01.798570 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2770
W20260810 15:20:01.814225 2745 connection.cc:535] client connection to 127.0.0.1:33663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33663): Connection reset by peer (error 104)
2026-08-10T15:20:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1497 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2948](info): set up 2 sockets
Aug 10 15:20:01 dist-test-slave-gjrz krb5kdc[2948](info): commencing operation
krb5kdc: starting...
W20260810 15:20:02.991509 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:03 dist-test-slave-gjrz krb5kdc[2948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:03.116176 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:03.199957 2972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:03.200201 2972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:03.200234 2972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:03.201444 2972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:03.201498 2972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:03.201516 2972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:03.201534 2972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:03.201547 2972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:03.203164 2972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:03.203464 2972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:03.203698 2972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:03.208530 2981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:03.208544 2980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:03.208778 2972 server_base.cc:1034] running on GCE node
W20260810 15:20:03.208595 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:03.209086 2972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:03.209391 2972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:03.210549 2972 hybrid_clock.cc:648] HybridClock initialized: now 1786375203210532 us; error 28 us; skew 500 ppm
Aug 10 15:20:03 dist-test-slave-gjrz krb5kdc[2948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:03.213225 2972 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:03.213650 2972 webserver.cc:460] Webserver started at http://127.0.0.1:44389/ using document root <none> and password file <none>
I20260810 15:20:03.213869 2972 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:03.213930 2972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:03.214067 2972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:03.215195 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "0957d597baa74e5fb06ba062ed09d947"
format_stamp: "Formatted at 2026-08-10 15:20:03 on dist-test-slave-gjrz"
server_key: "fb62439981b925d7608b6a414e81a8ba"
server_key_iv: "0ac2c43a49549a380028d170dca9f0bd"
server_key_version: "encryptionkey@0"
I20260810 15:20:03.215586 2972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "0957d597baa74e5fb06ba062ed09d947"
format_stamp: "Formatted at 2026-08-10 15:20:03 on dist-test-slave-gjrz"
server_key: "fb62439981b925d7608b6a414e81a8ba"
server_key_iv: "0ac2c43a49549a380028d170dca9f0bd"
server_key_version: "encryptionkey@0"
I20260810 15:20:03.216862 2972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:03.217584 2996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:03.217753 2972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:03.217839 2972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "0957d597baa74e5fb06ba062ed09d947"
format_stamp: "Formatted at 2026-08-10 15:20:03 on dist-test-slave-gjrz"
server_key: "fb62439981b925d7608b6a414e81a8ba"
server_key_iv: "0ac2c43a49549a380028d170dca9f0bd"
server_key_version: "encryptionkey@0"
I20260810 15:20:03.217908 2972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:03.230614 2972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:03.232731 2972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:03.232882 2972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:03.236953 2972 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44191
I20260810 15:20:03.237023 3093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44191 every 8 connection(s)
I20260810 15:20:03.237344 2972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:03.238308 3094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:03.240684 3094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: Bootstrap starting.
I20260810 15:20:03.241225 3094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:03.241466 3094 log.cc:824] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:03.242012 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 2972
I20260810 15:20:03.242120 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:03.242244 3094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: No bootstrap required, opened a new log
I20260810 15:20:03.242281 30616 external_mini_cluster.cc:1428] Setting key d14869b3ab930ffd4aa1406b64ab8290
I20260810 15:20:03.244758 3094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260810 15:20:03.244904 3094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:03.244940 3094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0957d597baa74e5fb06ba062ed09d947, State: Initialized, Role: FOLLOWER
I20260810 15:20:03.245069 3094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260810 15:20:03.245149 3094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:03.245186 3094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:03.245230 3094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:03.245960 3094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260810 15:20:03.246058 3094 leader_election.cc:302] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0957d597baa74e5fb06ba062ed09d947; no voters:
I20260810 15:20:03.246227 3094 leader_election.cc:288] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:03.246259 3103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:03.246411 3103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [term 1 LEADER]: Becoming Leader. State: Replica: 0957d597baa74e5fb06ba062ed09d947, State: Running, Role: LEADER
I20260810 15:20:03.246511 3094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:03.246572 3103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } }
I20260810 15:20:03.246982 3104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0957d597baa74e5fb06ba062ed09d947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260810 15:20:03.247079 3104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:03.247300 3105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0957d597baa74e5fb06ba062ed09d947. Latest consensus state: current_term: 1 leader_uuid: "0957d597baa74e5fb06ba062ed09d947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0957d597baa74e5fb06ba062ed09d947" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44191 } } }
I20260810 15:20:03.247409 3105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:03.248072 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:03.242787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:52223 (local address 127.0.0.1:44191)
0810 15:20:03.242944 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:03.242952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:03.242978 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:20:03.243520 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:03.243528 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:03.243580 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:03.243696 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:03.243929 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:03.244838 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:03.245885 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:03.246430 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:03.246860 (+ 430us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:03.247300 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:03.247304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:03.247305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:03.247506 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:03.247546 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:03.247562 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:03.247642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:03.247839 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 15:20:03.247869 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260810 15:20:03.248315 3131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:03.248373 3131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:03.248416 3120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:03.248569 3120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:03.249120 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:03.250089 3120 catalog_manager.cc:1347] Generated new cluster ID: c80c0265a0a94f61bb2c2670eb475b7c
I20260810 15:20:03.250142 3120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:03.253224 3101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:03.250498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:57735 (local address 127.0.0.1:44191)
0810 15:20:03.250535 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:03.250537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:03.250649 (+ 112us) server_negotiation.cc:407] Connection header received
0810 15:20:03.250711 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:03.250711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:03.250721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:03.250767 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:03.250882 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:03.251399 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:03.252100 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:03.252319 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:03.252374 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:03.252985 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:03.252987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:03.252987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:03.252991 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:03.253006 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:03.253010 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:03.253054 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:03.253144 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 15:20:03.253154 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:03.279559 3120 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:03.280172 3120 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:03.284463 3120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0957d597baa74e5fb06ba062ed09d947: Generated new TSK 0
I20260810 15:20:03.284682 3120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:03.296243 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 2972
W20260810 15:20:03.311081 2945 connection.cc:535] client connection to 127.0.0.1:44191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44191): Connection reset by peer (error 104)
2026-08-10T15:20:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:03 dist-test-slave-gjrz krb5kdc[3148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:03 dist-test-slave-gjrz krb5kdc[3148](info): set up 2 sockets
Aug 10 15:20:03 dist-test-slave-gjrz krb5kdc[3148](info): commencing operation
krb5kdc: starting...
W20260810 15:20:04.479156 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:04 dist-test-slave-gjrz krb5kdc[3148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:04.598316 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:04.679536 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:04.679744 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:04.679776 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:04.680891 3172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:04.680933 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:04.680953 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:04.680969 3172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:04.680984 3172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:04.682462 3172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:04.682698 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:04.682898 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:04.687703 3180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:04.687757 3179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:04.687709 3182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:04.687893 3172 server_base.cc:1034] running on GCE node
I20260810 15:20:04.688181 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:04.688457 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:04.689616 3172 hybrid_clock.cc:648] HybridClock initialized: now 1786375204689601 us; error 30 us; skew 500 ppm
Aug 10 15:20:04 dist-test-slave-gjrz krb5kdc[3148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:04.698105 3172 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:04.698527 3172 webserver.cc:460] Webserver started at http://127.0.0.1:36463/ using document root <none> and password file <none>
I20260810 15:20:04.698724 3172 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:04.698771 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:04.698894 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:04.699999 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "682209610f1f4427ba475787216d41a8"
format_stamp: "Formatted at 2026-08-10 15:20:04 on dist-test-slave-gjrz"
server_key: "15fcc796ba99488e37edd76007a90b5a"
server_key_iv: "63f256504a86ca2755dc57b1cec6617d"
server_key_version: "encryptionkey@0"
I20260810 15:20:04.700340 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "682209610f1f4427ba475787216d41a8"
format_stamp: "Formatted at 2026-08-10 15:20:04 on dist-test-slave-gjrz"
server_key: "15fcc796ba99488e37edd76007a90b5a"
server_key_iv: "63f256504a86ca2755dc57b1cec6617d"
server_key_version: "encryptionkey@0"
I20260810 15:20:04.701635 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:04.702423 3191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:04.702584 3172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:04.702673 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "682209610f1f4427ba475787216d41a8"
format_stamp: "Formatted at 2026-08-10 15:20:04 on dist-test-slave-gjrz"
server_key: "15fcc796ba99488e37edd76007a90b5a"
server_key_iv: "63f256504a86ca2755dc57b1cec6617d"
server_key_version: "encryptionkey@0"
I20260810 15:20:04.702735 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:04.706046 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:04.707989 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:04.708135 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:04.711980 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37861
I20260810 15:20:04.712045 3288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37861 every 8 connection(s)
I20260810 15:20:04.712330 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:04.713163 3289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:04.713253 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3172
I20260810 15:20:04.713370 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:04.713553 30616 external_mini_cluster.cc:1428] Setting key 3fd6edbc90b362a41dc7fd4a2d832170
I20260810 15:20:04.715840 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:04.714029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:40931 (local address 127.0.0.1:37861)
0810 15:20:04.714193 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:04.714201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:04.714226 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:20:04.714660 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:04.714667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:04.714692 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:04.714804 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:04.715431 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:04.715434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:04.715435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:04.715438 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:04.715479 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:04.715490 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:04.715544 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:04.715631 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 15:20:04.715664 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260810 15:20:04.716322 3289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8: Bootstrap starting.
I20260810 15:20:04.716696 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:04.716951 3289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:04.717198 3289 log.cc:824] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:04.717950 3289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8: No bootstrap required, opened a new log
I20260810 15:20:04.718989 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:04.718045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:52187 (local address 127.0.0.1:37861)
0810 15:20:04.718083 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:04.718085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:04.718182 (+ 97us) server_negotiation.cc:407] Connection header received
0810 15:20:04.718217 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:04.718218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:04.718222 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:04.718288 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:04.718747 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:04.718748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:04.718748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:04.718752 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:04.718766 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:04.718771 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:04.718817 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:04.718918 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 15:20:04.718930 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:20:04.720252 3289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20260810 15:20:04.720394 3289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:04.720417 3289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682209610f1f4427ba475787216d41a8, State: Initialized, Role: FOLLOWER
I20260810 15:20:04.720516 3289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20260810 15:20:04.720602 3289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:04.720640 3289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:04.720690 3289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:04.721226 3289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20260810 15:20:04.721318 3289 leader_election.cc:302] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 682209610f1f4427ba475787216d41a8; no voters:
I20260810 15:20:04.721481 3289 leader_election.cc:288] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:04.721550 3304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:04.721705 3304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [term 1 LEADER]: Becoming Leader. State: Replica: 682209610f1f4427ba475787216d41a8, State: Running, Role: LEADER
I20260810 15:20:04.721711 3289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:04.721877 3304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20260810 15:20:04.722357 3305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "682209610f1f4427ba475787216d41a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } } }
I20260810 15:20:04.722468 3305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:04.722663 3306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682209610f1f4427ba475787216d41a8. Latest consensus state: current_term: 1 leader_uuid: "682209610f1f4427ba475787216d41a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682209610f1f4427ba475787216d41a8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } } }
I20260810 15:20:04.722752 3306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:04.722704 3316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:04.723577 3316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:04.725205 3316 catalog_manager.cc:1347] Generated new cluster ID: a75cacbaaa134757941937ec5b388de6
I20260810 15:20:04.725248 3316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:04.746001 3316 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:04.746197 3316 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:04.751917 3316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 682209610f1f4427ba475787216d41a8: Generated new TSK 0
I20260810 15:20:04.752117 3316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:04.759104 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3172
2026-08-10T15:20:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:04 dist-test-slave-gjrz krb5kdc[3335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:04 dist-test-slave-gjrz krb5kdc[3335](info): set up 2 sockets
Aug 10 15:20:04 dist-test-slave-gjrz krb5kdc[3335](info): commencing operation
krb5kdc: starting...
W20260810 15:20:05.942680 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:06.066195 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:06.149912 3359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:06.150138 3359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:06.150170 3359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:06.151392 3359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:06.151439 3359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:06.151459 3359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:06.151475 3359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:06.151492 3359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:06.153017 3359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:06.153275 3359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:06.153486 3359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:06.158301 3366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:06.158464 3359 server_base.cc:1034] running on GCE node
W20260810 15:20:06.158293 3371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:06.158293 3368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:06.158797 3359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:06.159089 3359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:06.160238 3359 hybrid_clock.cc:648] HybridClock initialized: now 1786375206160209 us; error 48 us; skew 500 ppm
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:06.162770 3359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:06.163196 3359 webserver.cc:460] Webserver started at http://127.0.0.1:44467/ using document root <none> and password file <none>
I20260810 15:20:06.163417 3359 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:06.163466 3359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:06.163594 3359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:06.164669 3359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "4439ec4ffea5458081b35292e0c2cf1a"
format_stamp: "Formatted at 2026-08-10 15:20:06 on dist-test-slave-gjrz"
server_key: "8dca419f4969c4c0336a42eae1958f36"
server_key_iv: "44df3bc89ca4925fc68b8a0a48f79615"
server_key_version: "encryptionkey@0"
I20260810 15:20:06.165005 3359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "4439ec4ffea5458081b35292e0c2cf1a"
format_stamp: "Formatted at 2026-08-10 15:20:06 on dist-test-slave-gjrz"
server_key: "8dca419f4969c4c0336a42eae1958f36"
server_key_iv: "44df3bc89ca4925fc68b8a0a48f79615"
server_key_version: "encryptionkey@0"
I20260810 15:20:06.166252 3359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:06.166903 3382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:06.167071 3359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:06.167168 3359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "4439ec4ffea5458081b35292e0c2cf1a"
format_stamp: "Formatted at 2026-08-10 15:20:06 on dist-test-slave-gjrz"
server_key: "8dca419f4969c4c0336a42eae1958f36"
server_key_iv: "44df3bc89ca4925fc68b8a0a48f79615"
server_key_version: "encryptionkey@0"
I20260810 15:20:06.167241 3359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:06.179255 3359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:06.181406 3359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:06.181540 3359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:06.185575 3359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41751
I20260810 15:20:06.185626 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41751 every 8 connection(s)
I20260810 15:20:06.185946 3359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:06.186852 3476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:06.189232 3476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a: Bootstrap starting.
I20260810 15:20:06.189821 3476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:06.190083 3476 log.cc:824] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:06.190737 3476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a: No bootstrap required, opened a new log
I20260810 15:20:06.191834 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3359
I20260810 15:20:06.191926 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:06.192118 30616 external_mini_cluster.cc:1428] Setting key a7e06bb56343eeea194068c0cbbfa51c
I20260810 15:20:06.192430 3476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } }
I20260810 15:20:06.192610 3476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:06.192648 3476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4439ec4ffea5458081b35292e0c2cf1a, State: Initialized, Role: FOLLOWER
I20260810 15:20:06.192823 3476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } }
I20260810 15:20:06.192912 3476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:06.192952 3476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:06.192997 3476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:06.194073 3476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } }
I20260810 15:20:06.194188 3476 leader_election.cc:302] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4439ec4ffea5458081b35292e0c2cf1a; no voters:
I20260810 15:20:06.194342 3476 leader_election.cc:288] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:06.194380 3485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:06.194527 3485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [term 1 LEADER]: Becoming Leader. State: Replica: 4439ec4ffea5458081b35292e0c2cf1a, State: Running, Role: LEADER
I20260810 15:20:06.194599 3476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:06.194659 3485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } }
I20260810 15:20:06.195039 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4439ec4ffea5458081b35292e0c2cf1a. Latest consensus state: current_term: 1 leader_uuid: "4439ec4ffea5458081b35292e0c2cf1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } } }
I20260810 15:20:06.195300 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:06.195685 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:06.195197 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4439ec4ffea5458081b35292e0c2cf1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4439ec4ffea5458081b35292e0c2cf1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41751 } } }
I20260810 15:20:06.196383 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:06.196195 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:06.198137 3503 catalog_manager.cc:1347] Generated new cluster ID: 4ea4b5228ee549b5a82d4ef881bba710
I20260810 15:20:06.198189 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 15:20:06.203436 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:06.192612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:35949 (local address 127.0.0.1:41751)
0810 15:20:06.192800 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:06.192808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:06.192841 (+ 33us) server_negotiation.cc:407] Connection header received
0810 15:20:06.192954 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:06.192961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:06.193045 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:06.193204 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:06.193754 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:06.195157 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:06.196276 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:06.196826 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:06.196926 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:06.199696 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:06.199701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:06.199701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:06.199714 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:06.201646 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:06.202099 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:06.202109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:06.202110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:06.202248 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:06.202457 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:06.202458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:06.202458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:06.202805 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:06.202990 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:06.203072 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 15:20:06.203192 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260810 15:20:06.204538 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:06.214223 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:06.205906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55301 (local address 127.0.0.1:41751)
0810 15:20:06.205939 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:06.205941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:06.207416 (+ 1475us) server_negotiation.cc:407] Connection header received
0810 15:20:06.207449 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:06.207449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:06.207461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:06.207527 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:06.208097 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:06.208866 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:06.209843 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:06.210206 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:06.210268 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:06.211595 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:06.211596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:06.211597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:06.211605 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:06.213093 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:06.213474 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:06.213474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:06.213475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:06.213531 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:06.213719 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:06.213719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:06.213720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:06.213843 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:06.213956 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:06.214058 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 15:20:06.214129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:20:06.215972 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:06.216558 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:06.219936 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4439ec4ffea5458081b35292e0c2cf1a: Generated new TSK 0
I20260810 15:20:06.220113 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:06.225732 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3359
W20260810 15:20:06.239408 3333 connection.cc:535] client connection to 127.0.0.1:41751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41751): Connection reset by peer (error 104)
2026-08-10T15:20:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1464 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3545](info): set up 2 sockets
Aug 10 15:20:06 dist-test-slave-gjrz krb5kdc[3545](info): commencing operation
krb5kdc: starting...
W20260810 15:20:07.421511 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:07 dist-test-slave-gjrz krb5kdc[3545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:07.542747 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:07.624089 3564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:07.624316 3564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:07.624341 3564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:07.625502 3564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:07.625543 3564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:07.625557 3564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:07.625571 3564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:07.625594 3564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:07.627096 3564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:07.627421 3564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:07.627635 3564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:07.632336 3581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:07.632455 3564 server_base.cc:1034] running on GCE node
W20260810 15:20:07.632367 3577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:07.632551 3576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:07.632824 3564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:07.633106 3564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:07.634248 3564 hybrid_clock.cc:648] HybridClock initialized: now 1786375207634229 us; error 33 us; skew 500 ppm
Aug 10 15:20:07 dist-test-slave-gjrz krb5kdc[3545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:07.636636 3564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:07.637075 3564 webserver.cc:460] Webserver started at http://127.0.0.1:34065/ using document root <none> and password file <none>
I20260810 15:20:07.637296 3564 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:07.637341 3564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:07.637470 3564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:07.638494 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "4a456028e4d14890917c74c81d8d72bf"
format_stamp: "Formatted at 2026-08-10 15:20:07 on dist-test-slave-gjrz"
server_key: "f1787313de27ddc1bf6c787df7f8305c"
server_key_iv: "6ea66fee3022e1d0de64b1c7b83c5b28"
server_key_version: "encryptionkey@0"
I20260810 15:20:07.638826 3564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "4a456028e4d14890917c74c81d8d72bf"
format_stamp: "Formatted at 2026-08-10 15:20:07 on dist-test-slave-gjrz"
server_key: "f1787313de27ddc1bf6c787df7f8305c"
server_key_iv: "6ea66fee3022e1d0de64b1c7b83c5b28"
server_key_version: "encryptionkey@0"
I20260810 15:20:07.640122 3564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 15:20:07.640834 3592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:07.640991 3564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:07.641070 3564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "4a456028e4d14890917c74c81d8d72bf"
format_stamp: "Formatted at 2026-08-10 15:20:07 on dist-test-slave-gjrz"
server_key: "f1787313de27ddc1bf6c787df7f8305c"
server_key_iv: "6ea66fee3022e1d0de64b1c7b83c5b28"
server_key_version: "encryptionkey@0"
I20260810 15:20:07.641144 3564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:07.664609 3564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:07.666335 3564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:07.666440 3564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:07.670135 3564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32997
I20260810 15:20:07.670182 3697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32997 every 8 connection(s)
I20260810 15:20:07.670475 3564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:07.671375 3698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:07.673668 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf: Bootstrap starting.
I20260810 15:20:07.674233 3698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:07.674502 3698 log.cc:824] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:07.675208 3698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf: No bootstrap required, opened a new log
I20260810 15:20:07.676792 3698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260810 15:20:07.676931 3698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:07.676963 3698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a456028e4d14890917c74c81d8d72bf, State: Initialized, Role: FOLLOWER
I20260810 15:20:07.677114 3698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260810 15:20:07.677205 3698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:07.677238 3698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:07.677296 3698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:07.677681 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3564
I20260810 15:20:07.677783 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:07.677886 3698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260810 15:20:07.677964 30616 external_mini_cluster.cc:1428] Setting key db525939f40df7eb95465257ddd21a76
I20260810 15:20:07.677991 3698 leader_election.cc:302] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a456028e4d14890917c74c81d8d72bf; no voters:
I20260810 15:20:07.678158 3698 leader_election.cc:288] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:07.678196 3702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:07.678323 3702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [term 1 LEADER]: Becoming Leader. State: Replica: 4a456028e4d14890917c74c81d8d72bf, State: Running, Role: LEADER
I20260810 15:20:07.678428 3698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:07.678464 3702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } }
I20260810 15:20:07.679095 3705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a456028e4d14890917c74c81d8d72bf. Latest consensus state: current_term: 1 leader_uuid: "4a456028e4d14890917c74c81d8d72bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260810 15:20:07.679242 3705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:07.679731 3721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:07.679713 3703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a456028e4d14890917c74c81d8d72bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a456028e4d14890917c74c81d8d72bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32997 } } }
I20260810 15:20:07.679811 3703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:07.680241 3721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:07.682778 3721 catalog_manager.cc:1347] Generated new cluster ID: aba7e92036994d6aa8d918936cfe8ce3
I20260810 15:20:07.682822 3721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:07.683836 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:07.678563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:54439 (local address 127.0.0.1:32997)
0810 15:20:07.678854 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:07.678863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:07.678897 (+ 34us) server_negotiation.cc:407] Connection header received
0810 15:20:07.679016 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:07.679023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:07.679080 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:07.679506 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:07.679634 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:07.680868 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:07.681628 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:07.682232 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:07.682344 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:07.683132 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:07.683136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:07.683136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:07.683356 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:07.683393 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:07.683407 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:07.683485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:07.683613 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 15:20:07.683639 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20260810 15:20:07.692276 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:07.698823 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:07.693665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:34213 (local address 127.0.0.1:32997)
0810 15:20:07.693700 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:07.693702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:07.693830 (+ 128us) server_negotiation.cc:407] Connection header received
0810 15:20:07.693871 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:07.693872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:07.693883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:07.693932 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:07.694032 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:07.694803 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:07.695782 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:07.696081 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:07.696123 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:07.698565 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:07.698567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:07.698567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:07.698573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:07.698590 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:07.698595 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:07.698655 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:07.698732 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 15:20:07.698745 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:20:07.700398 3721 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:07.700883 3721 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:07.704263 3721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a456028e4d14890917c74c81d8d72bf: Generated new TSK 0
I20260810 15:20:07.704423 3721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:07.706404 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3564
W20260810 15:20:07.721169 3542 connection.cc:535] client connection to 127.0.0.1:32997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32997): Connection reset by peer (error 104)
2026-08-10T15:20:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:07 dist-test-slave-gjrz krb5kdc[3748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:07 dist-test-slave-gjrz krb5kdc[3748](info): set up 2 sockets
Aug 10 15:20:07 dist-test-slave-gjrz krb5kdc[3748](info): commencing operation
krb5kdc: starting...
W20260810 15:20:08.885622 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:08 dist-test-slave-gjrz krb5kdc[3748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 15:20:09.004041 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:09.086160 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:09.086360 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:09.086382 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:09.087510 3774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:09.087551 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:09.087564 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:09.087576 3774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:09.087594 3774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:09.089032 3774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:09.089222 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:09.089403 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:09.094067 3785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:09.094141 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:09.094090 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:09.094368 3774 server_base.cc:1034] running on GCE node
I20260810 15:20:09.094583 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:09.094857 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:09.095994 3774 hybrid_clock.cc:648] HybridClock initialized: now 1786375209095973 us; error 34 us; skew 500 ppm
Aug 10 15:20:09 dist-test-slave-gjrz krb5kdc[3748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:09.098349 3774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 15:20:09.098724 3774 webserver.cc:460] Webserver started at http://127.0.0.1:41931/ using document root <none> and password file <none>
I20260810 15:20:09.098912 3774 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:09.098950 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:09.099036 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:09.100025 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "58dab8e0aeef444f8d70aa916ed160f5"
format_stamp: "Formatted at 2026-08-10 15:20:09 on dist-test-slave-gjrz"
server_key: "c1c5dc05f39355d860ebc82a0e09965f"
server_key_iv: "84a45c64ad1a891cb441a3366cdd4eae"
server_key_version: "encryptionkey@0"
I20260810 15:20:09.100301 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "58dab8e0aeef444f8d70aa916ed160f5"
format_stamp: "Formatted at 2026-08-10 15:20:09 on dist-test-slave-gjrz"
server_key: "c1c5dc05f39355d860ebc82a0e09965f"
server_key_iv: "84a45c64ad1a891cb441a3366cdd4eae"
server_key_version: "encryptionkey@0"
I20260810 15:20:09.101435 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 15:20:09.102083 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:09.102246 3774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:09.102329 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "58dab8e0aeef444f8d70aa916ed160f5"
format_stamp: "Formatted at 2026-08-10 15:20:09 on dist-test-slave-gjrz"
server_key: "c1c5dc05f39355d860ebc82a0e09965f"
server_key_iv: "84a45c64ad1a891cb441a3366cdd4eae"
server_key_version: "encryptionkey@0"
I20260810 15:20:09.102381 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:09.105507 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:09.107434 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:09.107535 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:09.111128 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46321
I20260810 15:20:09.111163 3899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46321 every 8 connection(s)
I20260810 15:20:09.111469 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:09.112301 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:09.114370 3900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5: Bootstrap starting.
I20260810 15:20:09.114881 3900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:09.115109 3900 log.cc:824] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:09.115725 3900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5: No bootstrap required, opened a new log
I20260810 15:20:09.117357 3900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } }
I20260810 15:20:09.117486 3900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:09.117506 3900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dab8e0aeef444f8d70aa916ed160f5, State: Initialized, Role: FOLLOWER
I20260810 15:20:09.117609 3900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } }
I20260810 15:20:09.117681 3900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:09.117707 3900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:09.117738 3900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:09.118204 3900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } }
I20260810 15:20:09.118288 3900 leader_election.cc:302] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dab8e0aeef444f8d70aa916ed160f5; no voters:
I20260810 15:20:09.118424 3900 leader_election.cc:288] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:09.118460 3903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:09.118546 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3774
I20260810 15:20:09.118641 3903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [term 1 LEADER]: Becoming Leader. State: Replica: 58dab8e0aeef444f8d70aa916ed160f5, State: Running, Role: LEADER
I20260810 15:20:09.118669 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:09.118685 3900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:09.118736 3903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } }
I20260810 15:20:09.118865 30616 external_mini_cluster.cc:1428] Setting key ebeff62fd9b97ff24ac1e2002423bc75
I20260810 15:20:09.119076 3903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58dab8e0aeef444f8d70aa916ed160f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } } }
I20260810 15:20:09.119205 3903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:09.119405 3903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dab8e0aeef444f8d70aa916ed160f5. Latest consensus state: current_term: 1 leader_uuid: "58dab8e0aeef444f8d70aa916ed160f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dab8e0aeef444f8d70aa916ed160f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46321 } } }
I20260810 15:20:09.119509 3903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:09.120177 3916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:09.120661 3916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:09.121685 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:09.119535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:40837 (local address 127.0.0.1:46321)
0810 15:20:09.119721 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:09.119730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:09.120013 (+ 283us) server_negotiation.cc:407] Connection header received
0810 15:20:09.120149 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:09.120157 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:09.120186 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:09.120327 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:09.121288 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:09.121291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:09.121292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:09.121296 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:09.121336 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:09.121350 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:09.121419 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:09.121478 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 15:20:09.121520 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260810 15:20:09.122316 3916 catalog_manager.cc:1347] Generated new cluster ID: bb046d5f1bc34ac692db1767974d0f7f
I20260810 15:20:09.122363 3916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:09.122582 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:09.128475 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:09.124079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55751 (local address 127.0.0.1:46321)
0810 15:20:09.124117 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:09.124119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:09.127432 (+ 3313us) server_negotiation.cc:407] Connection header received
0810 15:20:09.127502 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:09.127503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:09.127508 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:09.127562 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:09.128235 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:09.128236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:09.128237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:09.128241 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:09.128258 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:09.128263 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:09.128328 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:09.128385 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 15:20:09.128398 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:09.135407 3916 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:09.135571 3916 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:09.137521 3916 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58dab8e0aeef444f8d70aa916ed160f5: Generated new TSK 0
I20260810 15:20:09.137689 3916 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:09.141158 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3774
2026-08-10T15:20:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:09 dist-test-slave-gjrz krb5kdc[3951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:09 dist-test-slave-gjrz krb5kdc[3951](info): set up 2 sockets
Aug 10 15:20:09 dist-test-slave-gjrz krb5kdc[3951](info): commencing operation
krb5kdc: starting...
W20260810 15:20:10.334333 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:10.450493 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42241
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:45355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:10.531101 3969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:10.531322 3969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:10.531343 3969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:10.532472 3969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:10.532514 3969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:10.532526 3969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:10.532537 3969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:10.532548 3969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:10.533977 3969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:45355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:10.534194 3969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:10.534396 3969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:10.539150 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:10.539139 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:10.539142 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:10.539192 3969 server_base.cc:1034] running on GCE node
I20260810 15:20:10.539574 3969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:10.539850 3969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:10.541014 3969 hybrid_clock.cc:648] HybridClock initialized: now 1786375210540991 us; error 31 us; skew 500 ppm
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375210, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:10.547612 3969 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:10.548019 3969 webserver.cc:460] Webserver started at http://127.29.230.62:36693/ using document root <none> and password file <none>
I20260810 15:20:10.548203 3969 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:10.548242 3969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:10.548334 3969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:10.549356 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "98cb57d71ea948348b6da044475e54a9"
format_stamp: "Formatted at 2026-08-10 15:20:10 on dist-test-slave-gjrz"
server_key: "144a7c055c1cdf7d48cb65cca753be8b"
server_key_iv: "d4d2b8ebe21a861aa1e4ccb3da2a18f1"
server_key_version: "encryptionkey@0"
I20260810 15:20:10.549650 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "98cb57d71ea948348b6da044475e54a9"
format_stamp: "Formatted at 2026-08-10 15:20:10 on dist-test-slave-gjrz"
server_key: "144a7c055c1cdf7d48cb65cca753be8b"
server_key_iv: "d4d2b8ebe21a861aa1e4ccb3da2a18f1"
server_key_version: "encryptionkey@0"
I20260810 15:20:10.550875 3969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:10.551649 3997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:10.551842 3969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 15:20:10.551927 3969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "98cb57d71ea948348b6da044475e54a9"
format_stamp: "Formatted at 2026-08-10 15:20:10 on dist-test-slave-gjrz"
server_key: "144a7c055c1cdf7d48cb65cca753be8b"
server_key_iv: "d4d2b8ebe21a861aa1e4ccb3da2a18f1"
server_key_version: "encryptionkey@0"
I20260810 15:20:10.551983 3969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:10.567170 3969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:10.568907 3969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:10.569028 3969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:10.572909 3969 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:42241
I20260810 15:20:10.572966 4091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:42241 every 8 connection(s)
I20260810 15:20:10.573264 3969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:10.574119 4092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:10.575559 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 3969
I20260810 15:20:10.575666 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:10.575842 30616 external_mini_cluster.cc:1428] Setting key 3e60562f7636f55762e14fe68d7994a1
I20260810 15:20:10.576738 4092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9: Bootstrap starting.
I20260810 15:20:10.577381 4092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:10.577739 4092 log.cc:824] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:10.578430 4092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9: No bootstrap required, opened a new log
I20260810 15:20:10.580680 4092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } }
I20260810 15:20:10.580825 4092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:10.580858 4092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cb57d71ea948348b6da044475e54a9, State: Initialized, Role: FOLLOWER
I20260810 15:20:10.580994 4092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } }
I20260810 15:20:10.581082 4092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:10.581115 4092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:10.581161 4092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:10.581856 4092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } }
I20260810 15:20:10.581986 4092 leader_election.cc:302] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98cb57d71ea948348b6da044475e54a9; no voters:
I20260810 15:20:10.582121 4092 leader_election.cc:288] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:10.582172 4105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:10.582278 4105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [term 1 LEADER]: Becoming Leader. State: Replica: 98cb57d71ea948348b6da044475e54a9, State: Running, Role: LEADER
I20260810 15:20:10.582348 4105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } }
I20260810 15:20:10.582371 4092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[3951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:10.582651 4105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98cb57d71ea948348b6da044475e54a9. Latest consensus state: current_term: 1 leader_uuid: "98cb57d71ea948348b6da044475e54a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } } }
I20260810 15:20:10.582712 4105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:10.582862 4106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98cb57d71ea948348b6da044475e54a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98cb57d71ea948348b6da044475e54a9" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42241 } } }
I20260810 15:20:10.582926 4106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:10.583041 4111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:10.583874 4111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:10.585847 4111 catalog_manager.cc:1347] Generated new cluster ID: 88a8fcc5eb25483c9230b82097f82673
I20260810 15:20:10.585978 4111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:10.587100 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:10.576351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38425 (local address 127.29.230.62:42241)
0810 15:20:10.576498 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:10.576506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:10.576531 (+ 25us) server_negotiation.cc:407] Connection header received
0810 15:20:10.576948 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:10.576955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:10.577003 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:10.577119 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:10.577901 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:10.578859 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:10.579931 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:10.580327 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:10.580403 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:10.583122 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:10.583126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:10.583126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:10.583139 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:10.585364 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:10.585774 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:10.585778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:10.585778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:10.585887 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:10.586167 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:10.586168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:10.586168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:10.586491 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:10.586630 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:10.586802 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 15:20:10.586899 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260810 15:20:10.588081 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:10.590132 4111 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:10.590835 4111 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:10.595288 4111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98cb57d71ea948348b6da044475e54a9: Generated new TSK 0
I20260810 15:20:10.595468 4111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:10.596149 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:10.589611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57333 (local address 127.29.230.62:42241)
0810 15:20:10.589644 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:10.589646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:10.589925 (+ 279us) server_negotiation.cc:407] Connection header received
0810 15:20:10.589946 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:10.589947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:10.589957 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:10.590008 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:10.590560 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:10.591040 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:10.591780 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:10.592006 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:10.592061 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:10.593423 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:10.593424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:10.593424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:10.593432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:10.594768 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:10.595259 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:10.595259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:10.595260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:10.595378 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:10.595644 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:10.595645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:10.595646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:10.595774 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:10.595888 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:10.595996 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 15:20:10.596059 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:20:10.598018 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 3969
W20260810 15:20:10.609818 3948 connection.cc:535] client connection to 127.29.230.62:42241 recv error: Network error: failed to read from TLS socket (remote: 127.29.230.62:42241): Connection reset by peer (error 104)
2026-08-10T15:20:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[4142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[4142](info): set up 2 sockets
Aug 10 15:20:10 dist-test-slave-gjrz krb5kdc[4142](info): commencing operation
krb5kdc: starting...
W20260810 15:20:11.785236 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:11 dist-test-slave-gjrz krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:11.904299 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36425
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:11.984969 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:11.985174 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:11.985196 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:11.986279 4162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:11.986315 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:11.986327 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:11.986339 4162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:11.986356 4162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:11.987815 4162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:36425
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:11.988010 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:11.988186 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:11.992818 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:11.992818 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:11.992909 4162 server_base.cc:1034] running on GCE node
W20260810 15:20:11.992841 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:11.993309 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:11.993593 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:11.994733 4162 hybrid_clock.cc:648] HybridClock initialized: now 1786375211994719 us; error 30 us; skew 500 ppm
Aug 10 15:20:11 dist-test-slave-gjrz krb5kdc[4142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375211, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:11.997140 4162 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:11.997602 4162 webserver.cc:460] Webserver started at http://127.29.230.62:42509/ using document root <none> and password file <none>
I20260810 15:20:11.997800 4162 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:11.997848 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:11.997973 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:11.999014 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "196089138976462f85ac673b22465057"
format_stamp: "Formatted at 2026-08-10 15:20:11 on dist-test-slave-gjrz"
server_key: "7912a8b228a87cd2b17db91bbe14cdb0"
server_key_iv: "86c524606212880b0161b5e10b5e7033"
server_key_version: "encryptionkey@0"
I20260810 15:20:11.999408 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "196089138976462f85ac673b22465057"
format_stamp: "Formatted at 2026-08-10 15:20:11 on dist-test-slave-gjrz"
server_key: "7912a8b228a87cd2b17db91bbe14cdb0"
server_key_iv: "86c524606212880b0161b5e10b5e7033"
server_key_version: "encryptionkey@0"
I20260810 15:20:12.000672 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:12.001368 4185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:12.001541 4162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 15:20:12.001626 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "196089138976462f85ac673b22465057"
format_stamp: "Formatted at 2026-08-10 15:20:11 on dist-test-slave-gjrz"
server_key: "7912a8b228a87cd2b17db91bbe14cdb0"
server_key_iv: "86c524606212880b0161b5e10b5e7033"
server_key_version: "encryptionkey@0"
I20260810 15:20:12.001693 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:12.030277 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:12.032199 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:12.032338 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:12.036327 4162 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:36425
I20260810 15:20:12.036371 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:36425 every 8 connection(s)
I20260810 15:20:12.036708 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:12.037582 4293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:12.039368 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4162
I20260810 15:20:12.039460 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:12.039649 30616 external_mini_cluster.cc:1428] Setting key 53388298028256f89b579331943ee79a
I20260810 15:20:12.040212 4293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: Bootstrap starting.
I20260810 15:20:12.041013 4293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:12.041359 4293 log.cc:824] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:12.042153 4293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: No bootstrap required, opened a new log
I20260810 15:20:12.044469 4293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } }
I20260810 15:20:12.044618 4293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:12.044652 4293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196089138976462f85ac673b22465057, State: Initialized, Role: FOLLOWER
I20260810 15:20:12.044777 4293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } }
I20260810 15:20:12.044838 4293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:12.044869 4293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:12.044916 4293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:12.045499 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:12.040162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36605 (local address 127.29.230.62:36425)
0810 15:20:12.040358 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:12.040368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:12.040400 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:20:12.041013 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:12.041024 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:12.041091 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:12.041255 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:12.041439 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:12.042758 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:12.043535 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:12.044179 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:12.044296 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:12.044910 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:12.044914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:12.044915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:12.045097 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:12.045129 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:12.045140 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:12.045212 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:12.045327 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 15:20:12.045354 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260810 15:20:12.045531 4293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } }
I20260810 15:20:12.045627 4293 leader_election.cc:302] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 196089138976462f85ac673b22465057; no voters:
I20260810 15:20:12.045796 4293 leader_election.cc:288] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:12.045842 4300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:12.045992 4300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [term 1 LEADER]: Becoming Leader. State: Replica: 196089138976462f85ac673b22465057, State: Running, Role: LEADER
I20260810 15:20:12.046064 4293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:12.046543 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:12.046083 4300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } }
I20260810 15:20:12.047801 4302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 196089138976462f85ac673b22465057. Latest consensus state: current_term: 1 leader_uuid: "196089138976462f85ac673b22465057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } } }
I20260810 15:20:12.047986 4302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:12.047811 4301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "196089138976462f85ac673b22465057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "196089138976462f85ac673b22465057" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 36425 } } }
I20260810 15:20:12.048205 4301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:12.048251 4329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:12.048295 4329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:12.048362 4337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:12.048571 4337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:12.050089 4337 catalog_manager.cc:1347] Generated new cluster ID: 2a084d479a9d4505b30a851c50575ec7
I20260810 15:20:12.050137 4337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:12.051848 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:12.048017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41359 (local address 127.29.230.62:36425)
0810 15:20:12.048050 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:12.048052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:12.048706 (+ 654us) server_negotiation.cc:407] Connection header received
0810 15:20:12.048802 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:12.048803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:12.048813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:12.048840 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:12.049038 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:12.049583 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:12.050561 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:12.050858 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:12.050920 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:12.051615 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:12.051616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:12.051616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:12.051621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:12.051635 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:12.051639 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:12.051694 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:12.051769 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 15:20:12.051778 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:20:12.058632 4337 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:12.059393 4337 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:12.067072 4337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 196089138976462f85ac673b22465057: Generated new TSK 0
I20260810 15:20:12.067318 4337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:12.072005 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4162
W20260810 15:20:12.087209 4140 connection.cc:535] client connection to 127.29.230.62:36425 recv error: Network error: failed to read from TLS socket (remote: 127.29.230.62:36425): Connection reset by peer (error 104)
2026-08-10T15:20:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1477 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:12 dist-test-slave-gjrz krb5kdc[4343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:12 dist-test-slave-gjrz krb5kdc[4343](info): set up 2 sockets
Aug 10 15:20:12 dist-test-slave-gjrz krb5kdc[4343](info): commencing operation
krb5kdc: starting...
W20260810 15:20:13.263782 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:13 dist-test-slave-gjrz krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:13.383391 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37645
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:37645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:13.464329 4364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:13.464543 4364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:13.464565 4364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:13.465724 4364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:13.465771 4364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:13.465785 4364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:13.465795 4364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:13.465806 4364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:13.467401 4364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:37645
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:37645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:13.467643 4364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:13.467859 4364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:13.472600 4370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:13.472649 4376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:13.472649 4372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:13.472797 4364 server_base.cc:1034] running on GCE node
I20260810 15:20:13.473151 4364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:13.473457 4364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:13.474607 4364 hybrid_clock.cc:648] HybridClock initialized: now 1786375213474584 us; error 50 us; skew 500 ppm
Aug 10 15:20:13 dist-test-slave-gjrz krb5kdc[4343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375213, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:13.477041 4364 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:13.477486 4364 webserver.cc:460] Webserver started at http://127.29.230.62:40821/ using document root <none> and password file <none>
I20260810 15:20:13.477702 4364 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:13.477752 4364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:13.477881 4364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:13.478951 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013"
format_stamp: "Formatted at 2026-08-10 15:20:13 on dist-test-slave-gjrz"
server_key: "16230a77071b582bc98a5c67ac2ee6c7"
server_key_iv: "5766d7523478eef8474b8cbd80abe23c"
server_key_version: "encryptionkey@0"
I20260810 15:20:13.479321 4364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013"
format_stamp: "Formatted at 2026-08-10 15:20:13 on dist-test-slave-gjrz"
server_key: "16230a77071b582bc98a5c67ac2ee6c7"
server_key_iv: "5766d7523478eef8474b8cbd80abe23c"
server_key_version: "encryptionkey@0"
I20260810 15:20:13.480607 4364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:13.481269 4384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:13.481424 4364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:13.481508 4364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013"
format_stamp: "Formatted at 2026-08-10 15:20:13 on dist-test-slave-gjrz"
server_key: "16230a77071b582bc98a5c67ac2ee6c7"
server_key_iv: "5766d7523478eef8474b8cbd80abe23c"
server_key_version: "encryptionkey@0"
I20260810 15:20:13.481580 4364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:13.484860 4364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:13.486611 4364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:13.486738 4364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:13.490671 4364 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:37645
I20260810 15:20:13.490708 4475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:37645 every 8 connection(s)
I20260810 15:20:13.491024 4364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:13.492278 4476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:13.495568 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: Bootstrap starting.
I20260810 15:20:13.496351 4476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:13.496692 4476 log.cc:824] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:13.498242 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4364
I20260810 15:20:13.498366 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:13.498559 30616 external_mini_cluster.cc:1428] Setting key 3c09205d2d317201e3a0764d8604cced
I20260810 15:20:13.501631 4476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: No bootstrap required, opened a new log
I20260810 15:20:13.503738 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:13.499042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51181 (local address 127.29.230.62:37645)
0810 15:20:13.499219 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:13.499229 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:13.501240 (+ 2011us) server_negotiation.cc:407] Connection header received
0810 15:20:13.501978 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:13.501987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:13.502022 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:13.502179 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:13.503161 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:13.503165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:13.503166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:13.503170 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:13.503215 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:13.503228 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:13.503337 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:13.503477 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 15:20:13.503525 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260810 15:20:13.504323 4476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } }
I20260810 15:20:13.504498 4476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:13.504546 4476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8bf4ab0a9c4ae4b419d19a3e119013, State: Initialized, Role: FOLLOWER
I20260810 15:20:13.504686 4476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } }
I20260810 15:20:13.504782 4476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:13.504815 4476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:13.504819 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:13.504860 4476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:13.505570 4476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } }
I20260810 15:20:13.505679 4476 leader_election.cc:302] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e8bf4ab0a9c4ae4b419d19a3e119013; no voters:
I20260810 15:20:13.505856 4476 leader_election.cc:288] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:13.505972 4485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:13.506147 4476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:13.506160 4485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [term 1 LEADER]: Becoming Leader. State: Replica: 7e8bf4ab0a9c4ae4b419d19a3e119013, State: Running, Role: LEADER
I20260810 15:20:13.506402 4485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } }
I20260810 15:20:13.507004 4490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e8bf4ab0a9c4ae4b419d19a3e119013. Latest consensus state: current_term: 1 leader_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } } }
I20260810 15:20:13.506985 4492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e8bf4ab0a9c4ae4b419d19a3e119013" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 37645 } } }
I20260810 15:20:13.507093 4490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:13.507097 4492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:13.507897 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:13.506300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46951 (local address 127.29.230.62:37645)
0810 15:20:13.506340 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:13.506341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:13.506571 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:20:13.506641 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:13.506642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:13.506646 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:13.506693 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:13.507653 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:13.507654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:13.507654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:13.507658 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:13.507673 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:13.507678 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:13.507725 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:13.507827 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 15:20:13.507841 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:20:13.508034 4519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:13.508085 4519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:13.508301 4512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:13.508525 4512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:13.510217 4512 catalog_manager.cc:1347] Generated new cluster ID: 2dc45235e8a64a1ba1804416b4d19d70
I20260810 15:20:13.510269 4512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:13.528798 4512 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:13.529013 4512 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:13.532467 4512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e8bf4ab0a9c4ae4b419d19a3e119013: Generated new TSK 0
I20260810 15:20:13.532644 4512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:13.537113 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4364
2026-08-10T15:20:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:13 dist-test-slave-gjrz krb5kdc[4531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:13 dist-test-slave-gjrz krb5kdc[4531](info): set up 2 sockets
Aug 10 15:20:13 dist-test-slave-gjrz krb5kdc[4531](info): commencing operation
krb5kdc: starting...
W20260810 15:20:14.719997 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:14 dist-test-slave-gjrz krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:14.841024 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:44375
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:39063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:14.923674 4553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:14.923877 4553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:14.923908 4553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:14.925045 4553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:14.925088 4553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:14.925107 4553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:14.925123 4553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:14.925138 4553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:14.926604 4553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:39063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:14.926848 4553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:14.927062 4553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:14.931878 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:14.932022 4553 server_base.cc:1034] running on GCE node
W20260810 15:20:14.931874 4566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:14.931874 4561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:14.932349 4553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:14.932646 4553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:14.933784 4553 hybrid_clock.cc:648] HybridClock initialized: now 1786375214933770 us; error 31 us; skew 500 ppm
Aug 10 15:20:14 dist-test-slave-gjrz krb5kdc[4531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375214, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:14.936210 4553 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:14.936645 4553 webserver.cc:460] Webserver started at http://127.29.230.62:36151/ using document root <none> and password file <none>
I20260810 15:20:14.936828 4553 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:14.936867 4553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:14.936955 4553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:14.937964 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a"
format_stamp: "Formatted at 2026-08-10 15:20:14 on dist-test-slave-gjrz"
server_key: "54890ca422a428668f59c82794337116"
server_key_iv: "fcce4b0a18f6a9a70a0ba4dcc3f7e90c"
server_key_version: "encryptionkey@0"
I20260810 15:20:14.938253 4553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a"
format_stamp: "Formatted at 2026-08-10 15:20:14 on dist-test-slave-gjrz"
server_key: "54890ca422a428668f59c82794337116"
server_key_iv: "fcce4b0a18f6a9a70a0ba4dcc3f7e90c"
server_key_version: "encryptionkey@0"
I20260810 15:20:14.939554 4553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:14.940215 4580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:14.940385 4553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:14.940521 4553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a"
format_stamp: "Formatted at 2026-08-10 15:20:14 on dist-test-slave-gjrz"
server_key: "54890ca422a428668f59c82794337116"
server_key_iv: "fcce4b0a18f6a9a70a0ba4dcc3f7e90c"
server_key_version: "encryptionkey@0"
I20260810 15:20:14.940586 4553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:14.957891 4553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:14.959779 4553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:14.959890 4553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:14.963644 4553 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:44375
I20260810 15:20:14.963694 4684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:44375 every 8 connection(s)
I20260810 15:20:14.964018 4553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:14.964890 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:14.966738 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4553
I20260810 15:20:14.966833 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:14.967000 30616 external_mini_cluster.cc:1428] Setting key 7ea3268e088e024ca573e20dbe195b3c
I20260810 15:20:14.967288 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a: Bootstrap starting.
I20260810 15:20:14.968047 4686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:14.968372 4686 log.cc:824] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:14.969163 4686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a: No bootstrap required, opened a new log
I20260810 15:20:14.971452 4686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } }
I20260810 15:20:14.971604 4686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:14.971654 4686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f905ffd58a14a7ea1b5deed9bd4ae3a, State: Initialized, Role: FOLLOWER
I20260810 15:20:14.971776 4686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } }
I20260810 15:20:14.971848 4686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:14.971884 4686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:14.971925 4686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:14.972574 4686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } }
I20260810 15:20:14.972697 4686 leader_election.cc:302] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f905ffd58a14a7ea1b5deed9bd4ae3a; no voters:
I20260810 15:20:14.972863 4686 leader_election.cc:288] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:14.972892 4694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:14.973014 4694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [term 1 LEADER]: Becoming Leader. State: Replica: 5f905ffd58a14a7ea1b5deed9bd4ae3a, State: Running, Role: LEADER
I20260810 15:20:14.973138 4686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:14.973167 4694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } }
I20260810 15:20:14.973626 4694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f905ffd58a14a7ea1b5deed9bd4ae3a. Latest consensus state: current_term: 1 leader_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } } }
I20260810 15:20:14.973851 4694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:14.973783 4695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f905ffd58a14a7ea1b5deed9bd4ae3a" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 44375 } } }
I20260810 15:20:14.974078 4695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:14.974429 4705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 15:20:14 dist-test-slave-gjrz krb5kdc[4531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:14.975083 4705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:14.978296 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:14.967531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57071 (local address 127.29.230.62:44375)
0810 15:20:14.967786 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:14.967795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:14.967827 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:20:14.968412 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:14.968422 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:14.968485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:14.968682 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:14.969397 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:14.970785 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:14.971605 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:14.972227 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:14.972330 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:14.975155 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:14.975159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:14.975160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:14.975172 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:14.976873 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:14.977204 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:14.977208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:14.977208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:14.977296 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:14.977473 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:14.977474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:14.977474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:14.977718 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:14.977863 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:14.977976 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 15:20:14.978112 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
I20260810 15:20:14.979297 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:14.979645 4705 catalog_manager.cc:1347] Generated new cluster ID: fbeb2dbe2e5f49e48c4f2f172038709c
I20260810 15:20:14.979699 4705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:14.985632 4705 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:14.986307 4705 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:14.990216 4705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f905ffd58a14a7ea1b5deed9bd4ae3a: Generated new TSK 0
I20260810 15:20:14.990378 4705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:14.992710 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:14.985742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47713 (local address 127.29.230.62:44375)
0810 15:20:14.985783 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:14.985785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:14.985987 (+ 202us) server_negotiation.cc:407] Connection header received
0810 15:20:14.986043 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:14.986044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:14.986054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:14.986112 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:14.986742 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:14.987478 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:14.988598 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:14.988923 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:14.988983 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:14.990409 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:14.990410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:14.990411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:14.990426 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:14.991711 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:14.992088 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:14.992088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:14.992088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:14.992136 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:14.992303 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:14.992303 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:14.992303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:14.992390 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:14.992461 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:14.992589 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 15:20:14.992654 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 15:20:14.994573 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4553
2026-08-10T15:20:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:15 dist-test-slave-gjrz krb5kdc[4738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:15 dist-test-slave-gjrz krb5kdc[4738](info): set up 2 sockets
Aug 10 15:20:15 dist-test-slave-gjrz krb5kdc[4738](info): commencing operation
krb5kdc: starting...
W20260810 15:20:16.174796 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.152s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:16 dist-test-slave-gjrz krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:16.294073 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40571
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:33025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:40571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:16.376233 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:16.376442 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:16.376466 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:16.377590 4757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:16.377633 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:16.377646 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:16.377657 4757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:16.377667 4757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:16.379105 4757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:33025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:40571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:40571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:16.379350 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:16.379565 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:16.384405 4757 server_base.cc:1034] running on GCE node
W20260810 15:20:16.384344 4772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:16.384375 4770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:16.384346 4768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:16.384858 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:16.385159 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:16.386324 4757 hybrid_clock.cc:648] HybridClock initialized: now 1786375216386309 us; error 28 us; skew 500 ppm
Aug 10 15:20:16 dist-test-slave-gjrz krb5kdc[4738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375216, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:16.393247 4757 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:16.393674 4757 webserver.cc:460] Webserver started at http://127.29.230.62:36349/ using document root <none> and password file <none>
I20260810 15:20:16.393869 4757 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:16.393918 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:16.394053 4757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:16.395104 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "14b25d3bbf29453ca842c5ba7e7076df"
format_stamp: "Formatted at 2026-08-10 15:20:16 on dist-test-slave-gjrz"
server_key: "8d9294436dd708f81de49aebba79d15c"
server_key_iv: "84fa53e45f34ac0602ab0c5797463f92"
server_key_version: "encryptionkey@0"
I20260810 15:20:16.395493 4757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "14b25d3bbf29453ca842c5ba7e7076df"
format_stamp: "Formatted at 2026-08-10 15:20:16 on dist-test-slave-gjrz"
server_key: "8d9294436dd708f81de49aebba79d15c"
server_key_iv: "84fa53e45f34ac0602ab0c5797463f92"
server_key_version: "encryptionkey@0"
I20260810 15:20:16.396766 4757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:16.397522 4782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:16.397717 4757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:16.397811 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "14b25d3bbf29453ca842c5ba7e7076df"
format_stamp: "Formatted at 2026-08-10 15:20:16 on dist-test-slave-gjrz"
server_key: "8d9294436dd708f81de49aebba79d15c"
server_key_iv: "84fa53e45f34ac0602ab0c5797463f92"
server_key_version: "encryptionkey@0"
I20260810 15:20:16.397876 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:16.412878 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:16.414999 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:16.415138 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:16.418952 4757 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:40571
I20260810 15:20:16.418993 4888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:40571 every 8 connection(s)
I20260810 15:20:16.419382 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:16.420233 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:16.422544 4889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df: Bootstrap starting.
I20260810 15:20:16.423151 4889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:16.423439 4889 log.cc:824] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:16.424129 4889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df: No bootstrap required, opened a new log
I20260810 15:20:16.425737 4889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } }
I20260810 15:20:16.425877 4889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:16.425899 4889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b25d3bbf29453ca842c5ba7e7076df, State: Initialized, Role: FOLLOWER
I20260810 15:20:16.426028 4889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } }
I20260810 15:20:16.426095 4889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:16.426115 4889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:16.426144 4889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:16.426949 4889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } }
I20260810 15:20:16.427040 4889 leader_election.cc:302] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14b25d3bbf29453ca842c5ba7e7076df; no voters:
I20260810 15:20:16.427186 4889 leader_election.cc:288] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:16.427232 4897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:16.427408 4897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [term 1 LEADER]: Becoming Leader. State: Replica: 14b25d3bbf29453ca842c5ba7e7076df, State: Running, Role: LEADER
I20260810 15:20:16.427446 4889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:16.427498 4897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } }
I20260810 15:20:16.427858 4899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14b25d3bbf29453ca842c5ba7e7076df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } } }
I20260810 15:20:16.427865 4900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14b25d3bbf29453ca842c5ba7e7076df. Latest consensus state: current_term: 1 leader_uuid: "14b25d3bbf29453ca842c5ba7e7076df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14b25d3bbf29453ca842c5ba7e7076df" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 40571 } } }
I20260810 15:20:16.428014 4899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:16.428025 4900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:16.428278 4910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:16.428802 4910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:16.429448 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4757
I20260810 15:20:16.429553 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:16.429738 30616 external_mini_cluster.cc:1428] Setting key a7b8be6947fd22d237ceb0c19053fb76
I20260810 15:20:16.430735 4910 catalog_manager.cc:1347] Generated new cluster ID: 16c53b7fa19e46bea8cab966a82bdc0d
I20260810 15:20:16.430781 4910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:16.436156 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:16.430291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47113 (local address 127.29.230.62:40571)
0810 15:20:16.430801 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:16.430810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:16.430842 (+ 32us) server_negotiation.cc:407] Connection header received
0810 15:20:16.430965 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:16.430985 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:16.431041 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:16.431177 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:16.431430 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:16.432931 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:16.433964 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:16.434599 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:16.434702 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:16.435353 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:16.435357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:16.435358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:16.435580 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:16.435624 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:16.435640 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:16.435714 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:16.435769 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 15:20:16.435935 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":66,"threads_started":1}
I20260810 15:20:16.437054 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:16.438334 4910 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:16.438913 4910 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:16.442147 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:16.438553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32807 (local address 127.29.230.62:40571)
0810 15:20:16.438593 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:16.438595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:16.438725 (+ 130us) server_negotiation.cc:407] Connection header received
0810 15:20:16.438770 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:16.438771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:16.438924 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:16.438985 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:16.439114 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:16.439828 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:16.440889 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:16.441177 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:16.441226 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:16.441932 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:16.441934 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:16.441934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:16.441938 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:16.441953 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:16.441957 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:16.442002 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:16.442051 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 15:20:16.442076 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:16.445312 4910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14b25d3bbf29453ca842c5ba7e7076df: Generated new TSK 0
I20260810 15:20:16.445492 4910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:16.447567 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4757
2026-08-10T15:20:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:16 dist-test-slave-gjrz krb5kdc[4939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:16 dist-test-slave-gjrz krb5kdc[4939](info): set up 2 sockets
Aug 10 15:20:16 dist-test-slave-gjrz krb5kdc[4939](info): commencing operation
krb5kdc: starting...
W20260810 15:20:17.635560 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:17 dist-test-slave-gjrz krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:17.756232 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33141
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:17.837801 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:17.838004 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:17.838025 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:17.839120 4960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:17.839155 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:17.839169 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:17.839180 4960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:17.839193 4960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:17.840608 4960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:17.840806 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:17.841040 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:17.845860 4969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:17.845963 4960 server_base.cc:1034] running on GCE node
W20260810 15:20:17.845885 4973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:17.845836 4968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:17.846297 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:17.846583 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:17.847710 4960 hybrid_clock.cc:648] HybridClock initialized: now 1786375217847696 us; error 31 us; skew 500 ppm
Aug 10 15:20:17 dist-test-slave-gjrz krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375217, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:17.850099 4960 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:17.850508 4960 webserver.cc:460] Webserver started at http://127.29.230.62:40483/ using document root <none> and password file <none>
I20260810 15:20:17.850708 4960 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:17.850759 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:17.850885 4960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:17.851996 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "ccbb8bbcf76b4ad4b9df4471828bd465"
format_stamp: "Formatted at 2026-08-10 15:20:17 on dist-test-slave-gjrz"
server_key: "c335ac2c37fd4bad16209ecba29300d8"
server_key_iv: "14956947753e82e814d1d9de2b65ba8e"
server_key_version: "encryptionkey@0"
I20260810 15:20:17.852339 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "ccbb8bbcf76b4ad4b9df4471828bd465"
format_stamp: "Formatted at 2026-08-10 15:20:17 on dist-test-slave-gjrz"
server_key: "c335ac2c37fd4bad16209ecba29300d8"
server_key_iv: "14956947753e82e814d1d9de2b65ba8e"
server_key_version: "encryptionkey@0"
I20260810 15:20:17.853621 4960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:17.854341 4983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:17.854543 4960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:17.854636 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "ccbb8bbcf76b4ad4b9df4471828bd465"
format_stamp: "Formatted at 2026-08-10 15:20:17 on dist-test-slave-gjrz"
server_key: "c335ac2c37fd4bad16209ecba29300d8"
server_key_iv: "14956947753e82e814d1d9de2b65ba8e"
server_key_version: "encryptionkey@0"
I20260810 15:20:17.854701 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:17.857959 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:17.859723 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:17.859849 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:17.863601 4960 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:33141
I20260810 15:20:17.863639 5073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:33141 every 8 connection(s)
I20260810 15:20:17.863925 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:17.864876 5074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:17.867100 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465: Bootstrap starting.
I20260810 15:20:17.867681 5074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:17.867952 5074 log.cc:824] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:17.868567 5074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465: No bootstrap required, opened a new log
I20260810 15:20:17.870160 5074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } }
I20260810 15:20:17.870294 5074 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 15:20:17.870316 5074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbb8bbcf76b4ad4b9df4471828bd465, State: Initialized, Role: FOLLOWER
I20260810 15:20:17.870445 5074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } }
I20260810 15:20:17.870508 5074 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 15:20:17.870532 5074 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 15:20:17.870575 5074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:17.871419 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 4960
I20260810 15:20:17.871456 5074 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } }
I20260810 15:20:17.871555 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:17.871564 5074 leader_election.cc:302] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbb8bbcf76b4ad4b9df4471828bd465; no voters:
I20260810 15:20:17.871719 5074 leader_election.cc:288] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 15:20:17.871762 30616 external_mini_cluster.cc:1428] Setting key e91f86061dd761873c0ab4e188b92af2
I20260810 15:20:17.871753 5079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:17.871892 5079 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [term 1 LEADER]: Becoming Leader. State: Replica: ccbb8bbcf76b4ad4b9df4471828bd465, State: Running, Role: LEADER
I20260810 15:20:17.871966 5074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:17.872035 5079 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } }
I20260810 15:20:17.872435 5085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbb8bbcf76b4ad4b9df4471828bd465. Latest consensus state: current_term: 1 leader_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } } }
I20260810 15:20:17.872493 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbb8bbcf76b4ad4b9df4471828bd465" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 33141 } } }
I20260810 15:20:17.872514 5085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:17.872539 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:17.873520 5089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:17.874027 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:17.872226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43651 (local address 127.29.230.62:33141)
0810 15:20:17.872376 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:17.872386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:17.872423 (+ 37us) server_negotiation.cc:407] Connection header received
0810 15:20:17.872573 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:17.872580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:17.872611 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:17.872798 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:17.873469 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:17.873472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:17.873473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:17.873476 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:17.873513 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:17.873525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:17.873774 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:17.873814 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 15:20:17.873855 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260810 15:20:17.874089 5089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:17.874967 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:17.877295 5089 catalog_manager.cc:1347] Generated new cluster ID: 01020f731d824466889877d4a2cea98b
I20260810 15:20:17.877341 5089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:17.886583 5089 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:17.886752 5089 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:17.887322 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:17.876514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57165 (local address 127.29.230.62:33141)
0810 15:20:17.876547 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:17.876548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:17.886404 (+ 9856us) server_negotiation.cc:407] Connection header received
0810 15:20:17.886463 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:17.886464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:17.886470 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:17.886517 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:17.887103 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:17.887104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:17.887104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 15:20:17.887109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:17.887125 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 15:20:17.887130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:17.887177 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:17.887230 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 15:20:17.887245 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:20:17.893317 5089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbb8bbcf76b4ad4b9df4471828bd465: Generated new TSK 0
I20260810 15:20:17.893497 5089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 15:20:17.898715 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 4960
2026-08-10T15:20:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1452 ms)
[----------] 18 tests from AuthTokenIssuingTest (26467 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:17 dist-test-slave-gjrz krb5kdc[5127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:17 dist-test-slave-gjrz krb5kdc[5127](info): set up 2 sockets
Aug 10 15:20:17 dist-test-slave-gjrz krb5kdc[5127](info): commencing operation
krb5kdc: starting...
W20260810 15:20:19.087713 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:19.210202 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:41653
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:19.291735 5147 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:19.291956 5147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:19.291981 5147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:19.292009 5147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:19.293164 5147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:19.293208 5147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:19.293226 5147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:19.293242 5147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:19.293257 5147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:19.295147 5147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:41653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:19.295423 5147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:19.295657 5147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:19.300326 5152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:19.300354 5153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:19.300457 5147 server_base.cc:1034] running on GCE node
W20260810 15:20:19.300328 5155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:19.300812 5147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:19.301090 5147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:19.302227 5147 hybrid_clock.cc:648] HybridClock initialized: now 1786375219302214 us; error 29 us; skew 500 ppm
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:19.304692 5147 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:19.305130 5147 webserver.cc:460] Webserver started at http://127.29.230.62:42465/ using document root <none> and password file <none>
I20260810 15:20:19.305328 5147 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:19.305377 5147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:19.305498 5147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:19.306547 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "a7f452eb87b04139be0ec7997631a9e1"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "eb4b4b185189a480b112520777b1dccf"
server_key_iv: "2ba7f6f52cdc598ec01c0ef32a85ad4f"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.306896 5147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "a7f452eb87b04139be0ec7997631a9e1"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "eb4b4b185189a480b112520777b1dccf"
server_key_iv: "2ba7f6f52cdc598ec01c0ef32a85ad4f"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.308274 5147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 15:20:19.309041 5165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:19.309235 5147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:19.309353 5147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "a7f452eb87b04139be0ec7997631a9e1"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "eb4b4b185189a480b112520777b1dccf"
server_key_iv: "2ba7f6f52cdc598ec01c0ef32a85ad4f"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.309422 5147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:19.351073 5147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:19.353037 5147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:19.353180 5147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:19.357196 5147 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:41653
I20260810 15:20:19.357236 5255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:41653 every 8 connection(s)
I20260810 15:20:19.357582 5147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:19.358603 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:19.360191 5257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.365736 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5147
I20260810 15:20:19.365839 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:19.366037 30616 external_mini_cluster.cc:1428] Setting key c16161327ba38eaa9b38782d5d9bf6e5
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:19.369817 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.360926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52386 (local address 127.29.230.62:41653)
0810 15:20:19.361098 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.361102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.361173 (+ 71us) server_negotiation.cc:407] Connection header received
0810 15:20:19.361672 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.361673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.361690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.361771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.362430 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.363350 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.364190 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.364503 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.364583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.366655 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.366663 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.366663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.366669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.368309 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.368683 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.368685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.368686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.368794 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.369023 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.369024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.369024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.369242 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.369306 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.369558 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.369657 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":100,"threads_started":1}
I20260810 15:20:19.369846 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.360845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:41653 (local address 127.0.0.1:52386)
0810 15:20:19.361052 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.361095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.361557 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.361782 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.361784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.362092 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.362304 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.362309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.363441 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.363442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.364115 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.364117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.364585 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.364586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.364637 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.365061 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.365070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.366562 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.368395 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.368402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.368404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.368606 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.368860 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.368860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.368861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.368957 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.369312 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.369320 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.369400 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.369511 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.369746 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":78,"thread_start_us":106,"threads_started":1}
I20260810 15:20:19.370553 5257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.370891 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.370760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52074)
0810 15:20:19.370809 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.370867 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:20:19.371064 5167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
I20260810 15:20:19.371506 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.371398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52088)
0810 15:20:19.371449 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.371483 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:20:19.371613 5257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:35959: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.29.230.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
I20260810 15:20:19.393455 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:35959
--webserver_interface=127.29.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:19.420830 5257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } attempt: 1
I20260810 15:20:19.421373 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.421216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52090)
0810 15:20:19.421279 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.421334 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:20:19.421721 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.421627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52106)
0810 15:20:19.421668 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.421698 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:20:19.421872 5257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:35959: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
W20260810 15:20:19.475806 5273 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:19.476023 5273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:19.476039 5273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:19.476058 5273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:19.477174 5273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:19.477216 5273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:19.477229 5273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:19.477239 5273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:19.477250 5273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:19.478781 5273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:19.479032 5273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:19.479241 5273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 15:20:19.483836 5273 server_base.cc:1034] running on GCE node
W20260810 15:20:19.483795 5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:19.483804 5279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:19.483794 5280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:19.484309 5273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:19.484604 5273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:19.485761 5273 hybrid_clock.cc:648] HybridClock initialized: now 1786375219485741 us; error 33 us; skew 500 ppm
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:19.488204 5273 init.cc:377] Logged in from keytab as kudu/127.29.230.61@KRBTEST.COM (short username kudu)
I20260810 15:20:19.488663 5273 webserver.cc:460] Webserver started at http://127.29.230.61:41403/ using document root <none> and password file <none>
I20260810 15:20:19.488886 5273 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:19.488933 5273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:19.489070 5273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:19.490036 5257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } attempt: 2
I20260810 15:20:19.490089 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data/instance:
uuid: "537921295ec34d50a321bc9e68f7d508"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "d3c0e72bfbcecff6d15b6953e17fc325"
server_key_iv: "e71c874407861fc7330736ba080eb687"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.490433 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.490292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52112)
0810 15:20:19.490355 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.490396 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:20:19.490486 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal/instance:
uuid: "537921295ec34d50a321bc9e68f7d508"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "d3c0e72bfbcecff6d15b6953e17fc325"
server_key_iv: "e71c874407861fc7330736ba080eb687"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.492244 5273 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 15:20:19.493131 5299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:19.493325 5273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 15:20:19.493413 5273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "537921295ec34d50a321bc9e68f7d508"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "d3c0e72bfbcecff6d15b6953e17fc325"
server_key_iv: "e71c874407861fc7330736ba080eb687"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.493484 5273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:19.494516 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.490674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52114)
0810 15:20:19.494436 (+ 3762us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.494472 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3737}
W20260810 15:20:19.494616 5257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:35959: Network error: Client connection negotiation failed: client connection to 127.29.230.61:35959: connect: Connection refused (error 111)
I20260810 15:20:19.505987 5273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:19.508100 5273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:19.508237 5273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:19.511993 5273 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.61:35959
I20260810 15:20:19.512033 5387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.61:35959 every 8 connection(s)
I20260810 15:20:19.512367 5273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
I20260810 15:20:19.513391 5391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:19.514878 5391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.518996 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5273
I20260810 15:20:19.519101 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-1/wal/instance
I20260810 15:20:19.519256 30616 external_mini_cluster.cc:1428] Setting key f9eacd01d1e4e5dcfb714379cb55e90f
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:19.523156 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.515586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.29.230.62:41653)
0810 15:20:19.515638 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.515640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.515850 (+ 210us) server_negotiation.cc:407] Connection header received
0810 15:20:19.516299 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.516300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.516312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.516361 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.516898 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.517385 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.518317 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.518537 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.518601 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.520623 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.520625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.520625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.520633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.521898 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.522279 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.522280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.522281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.522334 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.522555 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.522556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.522556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.522667 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.522722 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.522985 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.523067 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 15:20:19.523315 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.515572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:41653 (local address 127.0.0.1:52400)
0810 15:20:19.515739 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.515772 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.516228 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.516413 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.516416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.516644 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.516829 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.516833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.517454 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.517455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.518229 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.518231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.518622 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.518623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.518714 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.519212 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.519221 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.520537 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.521976 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.521985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.521988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.522212 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.522389 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.522390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.522390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.522499 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.522723 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.522730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.522862 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.522950 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.523095 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260810 15:20:19.523839 5391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.230.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.60@KRBTEST.COM" created.
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:19.533536 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.524085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52126)
0810 15:20:19.524123 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.524134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.524171 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.524456 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.524456 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.524573 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.524622 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.524623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.525833 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.525834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.526328 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.526329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.526992 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.526994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.527061 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.527468 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.527472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.531032 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.532588 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.532588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.532589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.532737 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.532908 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.532909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.532909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.532961 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.533208 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.533209 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.533247 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.533309 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.533420 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:20:19.533562 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.524146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52126 (local address 127.29.230.61:35959)
0810 15:20:19.524306 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.524311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.524333 (+ 22us) server_negotiation.cc:407] Connection header received
0810 15:20:19.524356 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.524358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.524372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.524454 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.524685 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.525753 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.526408 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.526889 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.526970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.531123 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.531141 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.531141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.531148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.532509 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.532807 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.532810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.532810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.532856 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.533009 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.533010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.533010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.533148 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.533218 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.533356 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.533471 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260810 15:20:19.534133 5391 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.534489 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.534343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41814)
0810 15:20:19.534399 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.534457 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 15:20:19.534634 5302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
I20260810 15:20:19.535044 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.534940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41824)
0810 15:20:19.534990 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.535021 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 15:20:19.535188 5391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:36097: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
I20260810 15:20:19.546057 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:36097
--webserver_interface=127.29.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:19.584471 5391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } attempt: 1
I20260810 15:20:19.585031 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.584877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41830)
0810 15:20:19.584936 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.584995 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:20:19.585371 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.585285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41844)
0810 15:20:19.585324 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.585351 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:19.585489 5391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:36097: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
I20260810 15:20:19.585743 5257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } attempt: 3
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:19.592943 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.586041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:52132)
0810 15:20:19.586103 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.586114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.586170 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.586303 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.586303 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.586467 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.586538 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.586540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.587535 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.587536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.588081 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.588083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.588573 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.588574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.588635 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.589059 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.589064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.590414 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.592100 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.592101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.592101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.592263 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.592441 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.592441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.592442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.592486 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.592718 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.592719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.592758 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.592802 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.592876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 15:20:19.592994 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.586091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.29.230.61:35959)
0810 15:20:19.586136 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.586138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.586158 (+ 20us) server_negotiation.cc:407] Connection header received
0810 15:20:19.586218 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.586219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.586232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.586295 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.586599 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.587435 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.588158 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.588503 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.588567 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.590497 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.590499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.590499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.590508 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.592025 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.592324 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.592325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.592325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.592386 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.592533 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.592534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.592535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.592667 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.592733 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.592825 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.592898 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:19.593683 5257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.594012 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.593922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41856)
0810 15:20:19.593963 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.593989 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:20:19.594370 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.594295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41858)
0810 15:20:19.594333 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.594355 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:19.594452 5257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:36097: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
I20260810 15:20:19.625633 5257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } attempt: 1
I20260810 15:20:19.626163 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.625991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41860)
0810 15:20:19.626071 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.626123 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 15:20:19.626492 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.626400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41864)
0810 15:20:19.626436 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.626468 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:20:19.626602 5257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:36097: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
W20260810 15:20:19.630635 5403 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:19.630854 5403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:19.630872 5403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:19.630908 5403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:19.632087 5403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:19.632130 5403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:19.632143 5403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:19.632155 5403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:19.632167 5403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:19.633659 5403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:41653,127.29.230.61:35959,127.29.230.60:36097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:19.633898 5403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:19.634115 5403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:19.638729 5413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:19.638728 5415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:19.638710 5411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:19.638749 5403 server_base.cc:1034] running on GCE node
I20260810 15:20:19.639130 5403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:19.639439 5403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:19.640578 5403 hybrid_clock.cc:648] HybridClock initialized: now 1786375219640560 us; error 29 us; skew 500 ppm
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:19.642916 5403 init.cc:377] Logged in from keytab as kudu/127.29.230.60@KRBTEST.COM (short username kudu)
I20260810 15:20:19.643359 5403 webserver.cc:460] Webserver started at http://127.29.230.60:32819/ using document root <none> and password file <none>
I20260810 15:20:19.643571 5403 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:19.643638 5403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:19.643774 5403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:19.645052 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data/instance:
uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "36359a4ea24e32d3c4ea4613b383f766"
server_key_iv: "84137c025a065130056a7768eefa9722"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.645354 5403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal/instance:
uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "36359a4ea24e32d3c4ea4613b383f766"
server_key_iv: "84137c025a065130056a7768eefa9722"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.646541 5403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 15:20:19.647163 5429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:19.647315 5403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:19.647397 5403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee"
format_stamp: "Formatted at 2026-08-10 15:20:19 on dist-test-slave-gjrz"
server_key: "36359a4ea24e32d3c4ea4613b383f766"
server_key_iv: "84137c025a065130056a7768eefa9722"
server_key_version: "encryptionkey@0"
I20260810 15:20:19.647459 5403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:19.653640 5391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } attempt: 2
I20260810 15:20:19.654021 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.653886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41870)
0810 15:20:19.653940 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.653984 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:20:19.654322 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.654253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:41884)
0810 15:20:19.654286 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.654307 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:20:19.654402 5391 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:36097: Network error: Client connection negotiation failed: client connection to 127.29.230.60:36097: connect: Connection refused (error 111)
I20260810 15:20:19.659447 5403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:19.661075 5403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:19.661181 5403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:19.664829 5403 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.60:36097
I20260810 15:20:19.664863 5520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.60:36097 every 8 connection(s)
I20260810 15:20:19.665180 5403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
I20260810 15:20:19.666133 5521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:19.667701 5521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:19.671258 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5403
I20260810 15:20:19.671370 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786374947162751-30616-0/minicluster-data/master-2/wal/instance
I20260810 15:20:19.671566 30616 external_mini_cluster.cc:1428] Setting key 1c1fb064886418f9eec06c3999a9dd4c
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:19.676604 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.668373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.29.230.62:41653)
0810 15:20:19.668425 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.668427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.668624 (+ 197us) server_negotiation.cc:407] Connection header received
0810 15:20:19.669066 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.669067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.669075 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.669142 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.669710 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.670180 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.671128 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.671345 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.671406 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.673925 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.673926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.673927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.673933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.675237 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.675668 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.675669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.675669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.675724 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.676002 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.676003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.676004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.676138 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.676212 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.676430 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.676512 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 15:20:19.676674 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.668338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:41653 (local address 127.0.0.1:52412)
0810 15:20:19.668516 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.668555 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.668993 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.669204 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.669207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.669439 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.669644 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.669648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.670245 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.670246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.671038 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.671040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.671486 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.671487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.671652 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.672167 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.672176 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.673845 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.675314 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.675319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.675321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.675600 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.675781 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.675781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.675782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.675914 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.676202 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.676206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.676314 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.676392 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.676504 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260810 15:20:19.677225 5521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:19.680523 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.671944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.29.230.60:36097)
0810 15:20:19.672111 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.672116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.672221 (+ 105us) server_negotiation.cc:407] Connection header received
0810 15:20:19.672277 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.672279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.672297 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.672354 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.673044 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.674054 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.675039 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.675388 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.675457 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.677768 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.677777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.677777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.677784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.679353 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.679679 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.679682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.679682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.679738 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.679930 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.679931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.679932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.680099 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.680208 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.680331 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.680407 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:19.683769 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.677595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.29.230.61:35959)
0810 15:20:19.677637 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.677638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.677665 (+ 27us) server_negotiation.cc:407] Connection header received
0810 15:20:19.677687 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.677688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.677697 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.677742 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.677978 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.678615 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.679297 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.679567 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.679618 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.681484 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.681485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.681486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.681493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.682805 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.683106 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.683107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.683107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.683164 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.683331 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.683331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.683332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.683447 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.683514 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.683648 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.683710 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:19.683800 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.677534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:35959 (local address 127.0.0.1:48114)
0810 15:20:19.677577 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.677586 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.677620 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.677759 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.677760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.677867 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.677910 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.677911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.678719 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.678720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.679190 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.679192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.679632 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.679633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.679711 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.680035 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.680039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.681412 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.682873 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.682874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.682874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.683023 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.683221 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.683221 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.683222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.683288 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.683512 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.683513 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.683555 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.683628 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.683712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:20:19.684216 5521 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:19.688786 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.681364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.29.230.62:41653)
0810 15:20:19.681403 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.681405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.681448 (+ 43us) server_negotiation.cc:407] Connection header received
0810 15:20:19.681518 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.681519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.681528 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.681567 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.682109 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.682627 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.683526 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.683793 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.683851 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.686318 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.686319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.686319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.686325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.687698 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.688034 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.688034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.688035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.688087 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.688286 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.688287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.688287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.688410 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.688565 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.688629 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.688700 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:19.690680 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.684485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:51556)
0810 15:20:19.684517 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.684526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.684562 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.684679 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.684680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.684782 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.684810 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.684811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.685575 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.685576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.686315 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.686317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.686746 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.686747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.686786 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.687102 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.687106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.688264 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.689791 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.689792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.689792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.689939 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.690112 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.690112 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.690113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.690169 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.690363 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.690364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.690405 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.690475 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.690564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 15:20:19.690680 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.684503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.29.230.60:36097)
0810 15:20:19.684536 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.684538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.684550 (+ 12us) server_negotiation.cc:407] Connection header received
0810 15:20:19.684615 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.684616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.684625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.684677 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.684866 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.685501 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.686386 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.686676 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.686739 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.688333 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.688334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.688334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.688341 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.689726 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.689998 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.689999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.689999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.690055 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.690221 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.690221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.690222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.690307 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.690350 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.690492 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.690603 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":17}
I20260810 15:20:19.692171 5521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: Bootstrap starting.
I20260810 15:20:19.693179 5521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:19.693468 5521 log.cc:824] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: Log is configured to *not* fsync() on all Append() calls
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:19.694224 5521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: No bootstrap required, opened a new log
I20260810 15:20:19.695696 5521 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.695822 5521 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc7f5dde4cd4115b70cd2e20f1abeee, State: Initialized, Role: FOLLOWER
I20260810 15:20:19.695895 5521 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.696135 5529 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.696265 5529 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.696199 5521 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:19.696408 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.689441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.29.230.61:35959)
0810 15:20:19.689475 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.689477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.689484 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:19.689553 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.689554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.689566 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.689608 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.690102 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.690777 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.691565 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.691847 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.691923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.694114 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.694115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.694115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.694122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.695382 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.695670 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.695671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.695671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.695722 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.695946 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.695947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.695947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.696069 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.696224 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.696258 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.696324 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 15:20:19.697889 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 15:20:19.698194 5546 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:19.698251 5546 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:19.699098 5211 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7f452eb87b04139be0ec7997631a9e1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57418
I20260810 15:20:19.698479 5403 master_runner.cc:186] Error getting master registration for 127.29.230.62:41653: OK, instance_id { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" instance_seqno: 1786375219356070 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:20:19.700551 5403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:20:19.701748 5257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } attempt: 2
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:19.708844 5260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.702025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:51562)
0810 15:20:19.702067 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.702080 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.702135 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.702272 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.702273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.702431 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.702486 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.702488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.703354 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.703354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.704165 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.704166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.704609 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.704610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.704662 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.705047 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.705052 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.706268 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.707709 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.707710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.707710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.707926 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.708096 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.708097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.708097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.708184 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.708520 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.708521 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.708585 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.708661 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.708757 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260810 15:20:19.708850 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.702034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.29.230.60:36097)
0810 15:20:19.702093 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.702094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.702186 (+ 92us) server_negotiation.cc:407] Connection header received
0810 15:20:19.702207 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.702208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.702217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.702267 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.702556 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.703258 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.704241 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.704531 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.704612 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.706348 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.706349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.706349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.706355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.707642 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.707993 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.707994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.707994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.708041 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.708256 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.708257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.708258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.708419 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.708488 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.708706 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.708779 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 15:20:19.710615 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: Bootstrap starting.
I20260810 15:20:19.711387 5257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:19.711704 5257 log.cc:824] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:19.712648 5257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: No bootstrap required, opened a new log
I20260810 15:20:19.713965 5257 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.714111 5257 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f452eb87b04139be0ec7997631a9e1, State: Initialized, Role: FOLLOWER
I20260810 15:20:19.714191 5257 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.714479 5553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.714574 5553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.714586 5257 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:19.714622 5211 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:20:19.714700 5211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:19.715376 5211 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.715571 5211 leader_election.cc:288] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [CANDIDATE]: Term 1 election: Requested vote from peers 537921295ec34d50a321bc9e68f7d508 (127.29.230.61:35959), bfc7f5dde4cd4115b70cd2e20f1abeee (127.29.230.60:36097)
I20260810 15:20:19.716223 5478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f452eb87b04139be0ec7997631a9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee"
I20260810 15:20:19.716367 5478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [term 0 FOLLOWER]: Advancing to term 1
W20260810 15:20:19.716470 5573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:19.716516 5573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:19.717242 5478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7f452eb87b04139be0ec7997631a9e1 in term 1.
I20260810 15:20:19.717541 5167 leader_election.cc:302] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7f452eb87b04139be0ec7997631a9e1, bfc7f5dde4cd4115b70cd2e20f1abeee; no voters:
I20260810 15:20:19.717659 5553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:19.717787 5553 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [term 1 LEADER]: Becoming Leader. State: Replica: a7f452eb87b04139be0ec7997631a9e1, State: Running, Role: LEADER
I20260810 15:20:19.717878 5553 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.718298 5553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f452eb87b04139be0ec7997631a9e1. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.718369 5553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:19.718647 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:19.718935 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:19.717085 5147 master_runner.cc:186] Error getting master registration for 127.29.230.61:35959: OK, instance_id { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" instance_seqno: 1786375219510874 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:20:19.719950 5147 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:20:19.721514 5478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [term 1 FOLLOWER]: Refusing update from remote peer a7f452eb87b04139be0ec7997631a9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:19.721868 5585 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:19.723299 5529 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f452eb87b04139be0ec7997631a9e1. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.723407 5529 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.724336 5553 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.724412 5553 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:19.724627 5592 mvcc.cc:205] Tried to move back new op lower bound from 7316992899976564736 to 7316992899965177856. Current Snapshot: MvccSnapshot[applied={T|T < 7316992899976564736}]
I20260810 15:20:19.724881 5529 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.724957 5529 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.725168 5595 mvcc.cc:205] Tried to move back new op lower bound from 7316992899976564736 to 7316992899965177856. Current Snapshot: MvccSnapshot[applied={T|T < 7316992899976564736}]
I20260810 15:20:19.728508 5591 catalog_manager.cc:1347] Generated new cluster ID: b1e92436739c483884d7157d50e8c47b
I20260810 15:20:19.728562 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:19.730429 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.716844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57430 (local address 127.29.230.62:41653)
0810 15:20:19.717383 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.717386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.718231 (+ 845us) server_negotiation.cc:407] Connection header received
0810 15:20:19.718328 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.718329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.718339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.718367 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.720456 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.721119 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.724103 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.724370 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.724411 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.727567 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.727568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.727568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.727575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.728956 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.729507 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.729508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.729509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.729571 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.729798 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.729799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.729799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.729924 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.730003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.730274 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.730345 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515}
I20260810 15:20:19.736192 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.717474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.29.230.61:35959)
0810 15:20:19.717541 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.717543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.718858 (+ 1315us) server_negotiation.cc:407] Connection header received
0810 15:20:19.719100 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.719101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.719113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.719143 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.721657 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.722356 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.724777 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.725072 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.725117 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.727397 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.727398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.727398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.727405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.734489 (+ 7084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.735082 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.735084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.735084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.735149 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.735469 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.735471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.735471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.735628 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.735711 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.736010 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.736097 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 15:20:19.739812 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.719330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.29.230.60:36097)
0810 15:20:19.719370 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.719372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.731382 (+ 12010us) server_negotiation.cc:407] Connection header received
0810 15:20:19.731482 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.731484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.731505 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.731533 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.732249 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.732951 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.734262 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.734568 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.734605 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.736667 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.736669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.736669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.736675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.738315 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.738804 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.738805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.738805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.738874 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.739121 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.739122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.739122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.739245 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.739371 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.739654 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.739726 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 15:20:19.745061 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:19.745537 5391 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } attempt: 3
I20260810 15:20:19.745867 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:19.749725 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1: Generated new TSK 0
I20260810 15:20:19.749907 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
Aug 10 15:20:19 dist-test-slave-gjrz krb5kdc[5127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375219, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:19.759126 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.745805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:36097 (local address 127.0.0.1:51576)
0810 15:20:19.745891 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:19.745901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:19.745937 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:19.746807 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:19.746808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:19.746980 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:19.747029 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.747031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.747972 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.747973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.748668 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:19.748671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.749443 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.749444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:19.749514 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.750207 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:19.750213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:19.751764 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:19.755194 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.755195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.755210 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.755447 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.758472 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:19.758473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:19.758473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:19.758543 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:19.758814 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:19.758815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:19.758863 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 15:20:19.758908 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 15:20:19.759015 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 15:20:19.759773 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:19.745909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.29.230.60:36097)
0810 15:20:19.746690 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.746692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.746699 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:19.746730 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.746731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.746740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.746783 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.747107 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.747867 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.748815 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:19.749288 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:19.749397 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:19.751853 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:19.751854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:19.751855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:19.751862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:19.755128 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.758316 (+ 3188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.758318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.758318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.758388 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:19.758613 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:19.758613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:19.758614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:19.758742 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:19.759560 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:19.759602 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:20:19.759680 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762}
I20260810 15:20:19.761276 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: Bootstrap starting.
I20260810 15:20:19.762032 5391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:19.762343 5391 log.cc:824] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:19.763294 5391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: No bootstrap required, opened a new log
I20260810 15:20:19.764715 5391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.764832 5391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537921295ec34d50a321bc9e68f7d508, State: Initialized, Role: FOLLOWER
I20260810 15:20:19.764906 5391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } }
I20260810 15:20:19.765152 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.765224 5391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:19.765321 5342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:19.766242 5342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [term 1 FOLLOWER]: Refusing update from remote peer a7f452eb87b04139be0ec7997631a9e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:19.766443 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7f452eb87b04139be0ec7997631a9e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "537921295ec34d50a321bc9e68f7d508"
I20260810 15:20:19.766662 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.767187 5344 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7f452eb87b04139be0ec7997631a9e1 in term 1.
I20260810 15:20:19.767549 5553 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 15:20:19.768020 5624 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:19.768071 5624 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:19.769263 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7f452eb87b04139be0ec7997631a9e1. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.769378 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.769447 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:19.769533 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:19.771622 5627 mvcc.cc:205] Tried to move back new op lower bound from 7316992900088979456 to 7316992899965177856. Current Snapshot: MvccSnapshot[applied={T|T < 7316992899976564736}]
W20260810 15:20:19.787110 5264 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.785394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.29.230.62:41653)
0810 15:20:19.785442 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.785444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.786388 (+ 944us) server_negotiation.cc:407] Connection header received
0810 15:20:19.786492 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.786493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.786504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.786533 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.786985 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57432: BlockingRecv error: recv got EOF from 127.0.0.1:57432 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 15:20:19.787930 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.785782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.29.230.61:35959)
0810 15:20:19.785823 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.785825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.787415 (+ 1590us) server_negotiation.cc:407] Connection header received
0810 15:20:19.787515 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.787516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.787527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.787558 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.787870 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48132: BlockingRecv error: recv got EOF from 127.0.0.1:48132 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:20:19.788645 5527 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.786095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.29.230.60:36097)
0810 15:20:19.786135 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.786137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.788161 (+ 2024us) server_negotiation.cc:407] Connection header received
0810 15:20:19.788250 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.788251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.788261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.788288 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.788580 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51592: BlockingRecv error: recv got EOF from 127.0.0.1:51592 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:20:19.792616 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.791843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.29.230.61:35959)
0810 15:20:19.791888 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.791890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.792114 (+ 224us) server_negotiation.cc:407] Connection header received
0810 15:20:19.792219 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.792219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.792240 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.792268 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.792568 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48136: BlockingRecv error: recv got EOF from 127.0.0.1:48136 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:20:19.795616 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.794833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.29.230.61:35959)
0810 15:20:19.794863 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.794864 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.795089 (+ 225us) server_negotiation.cc:407] Connection header received
0810 15:20:19.795181 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.795182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.795192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.795220 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.795574 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48152: BlockingRecv error: recv got EOF from 127.0.0.1:48152 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260810 15:20:19.799516 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.798765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.29.230.61:35959)
0810 15:20:19.798798 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.798800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.798996 (+ 196us) server_negotiation.cc:407] Connection header received
0810 15:20:19.799096 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.799097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.799107 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.799133 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.799474 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48160: BlockingRecv error: recv got EOF from 127.0.0.1:48160 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 15:20:19.805305 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.804589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.29.230.61:35959)
0810 15:20:19.804622 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.804624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.804827 (+ 203us) server_negotiation.cc:407] Connection header received
0810 15:20:19.804911 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.804911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.804920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.804944 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.805265 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48176: BlockingRecv error: recv got EOF from 127.0.0.1:48176 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 15:20:19.815479 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.814536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.29.230.61:35959)
0810 15:20:19.814579 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.814582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.814852 (+ 270us) server_negotiation.cc:407] Connection header received
0810 15:20:19.814948 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.814949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.814962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.814990 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.815409 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48186: BlockingRecv error: recv got EOF from 127.0.0.1:48186 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:20:19.833928 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.832902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.29.230.61:35959)
0810 15:20:19.832949 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.832951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.833270 (+ 319us) server_negotiation.cc:407] Connection header received
0810 15:20:19.833397 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.833398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.833411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.833444 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.833869 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48198: BlockingRecv error: recv got EOF from 127.0.0.1:48198 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:20:19.868295 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.867347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.29.230.61:35959)
0810 15:20:19.867393 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.867395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.867683 (+ 288us) server_negotiation.cc:407] Connection header received
0810 15:20:19.867782 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.867783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.867796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.867823 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.868232 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48204: BlockingRecv error: recv got EOF from 127.0.0.1:48204 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 15:20:19.934260 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:19.933615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.29.230.61:35959)
0810 15:20:19.933669 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:19.933671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:19.933730 (+ 59us) server_negotiation.cc:407] Connection header received
0810 15:20:19.933790 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:19.933790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:19.933802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:19.933880 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:19.934201 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48210: BlockingRecv error: recv got EOF from 127.0.0.1:48210 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 15:20:20.064101 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:20.063408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.29.230.61:35959)
0810 15:20:20.063479 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.063481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.063542 (+ 61us) server_negotiation.cc:407] Connection header received
0810 15:20:20.063599 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.063600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.063610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.063675 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:20.064042 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48216: BlockingRecv error: recv got EOF from 127.0.0.1:48216 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 15:20:20.317158 5731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7f452eb87b04139be0ec7997631a9e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7f452eb87b04139be0ec7997631a9e1" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 41653 } } peers { permanent_uuid: "537921295ec34d50a321bc9e68f7d508" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 35959 } } peers { permanent_uuid: "bfc7f5dde4cd4115b70cd2e20f1abeee" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 36097 } } }
I20260810 15:20:20.317317 5731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7f452eb87b04139be0ec7997631a9e1 [sys.catalog]: This master's current role is: LEADER
W20260810 15:20:20.321949 5397 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:20.321339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.29.230.61:35959)
0810 15:20:20.321408 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.321409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.321419 (+ 10us) server_negotiation.cc:407] Connection header received
0810 15:20:20.321499 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.321500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.321512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.321556 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:20.321899 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48226: BlockingRecv error: recv got EOF from 127.0.0.1:48226 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 15:20:20.698696 5546 catalog_manager.cc:1259] Loaded cluster ID: b1e92436739c483884d7157d50e8c47b
I20260810 15:20:20.698768 5546 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: loading cluster ID for follower catalog manager: success
I20260810 15:20:20.699646 5546 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: acquiring CA information for follower catalog manager: success
I20260810 15:20:20.699967 5546 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfc7f5dde4cd4115b70cd2e20f1abeee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:20:20.768538 5624 catalog_manager.cc:1259] Loaded cluster ID: b1e92436739c483884d7157d50e8c47b
I20260810 15:20:20.768617 5624 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: loading cluster ID for follower catalog manager: success
I20260810 15:20:20.769477 5624 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: acquiring CA information for follower catalog manager: success
I20260810 15:20:20.769882 5624 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 537921295ec34d50a321bc9e68f7d508: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:20:20.837818 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:20.835202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.29.230.61:35959)
0810 15:20:20.835329 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.835331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.835395 (+ 64us) server_negotiation.cc:407] Connection header received
0810 15:20:20.835457 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.835458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.835517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.835585 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:20.835729 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.836232 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.837041 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.837240 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.837297 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:20.837466 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:20.837585 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:20.837653 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:20.837703 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 15:20:20.837728 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260810 15:20:20.842011 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:20.839715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.29.230.60:36097)
0810 15:20:20.839822 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.839824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.839835 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:20.839900 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.839901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.839935 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.840008 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:20.840125 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.840597 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.841293 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.841497 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.841546 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:20.841663 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:20.841767 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:20.841821 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:20.841918 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 15:20:20.841934 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260810 15:20:20.846220 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:20.843806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.29.230.61:35959)
0810 15:20:20.843851 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.843852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.843970 (+ 118us) server_negotiation.cc:407] Connection header received
0810 15:20:20.844069 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.844070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.844080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.844108 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:20.844294 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.844766 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.845574 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.845779 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.845840 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:20.845990 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:20.846022 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:20.846070 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:20.846126 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:20:20.846147 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 15:20:20.847013 5772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:20.843673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57434 (local address 127.29.230.62:41653)
0810 15:20:20.843781 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.843783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.843791 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:20.843848 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.843849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.843901 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.843999 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:20.844161 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.844701 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.845824 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.846127 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.846180 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:20.846402 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:20.846512 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:20.846570 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:20.846925 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 15:20:20.846945 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260810 15:20:20.847386 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:20.844078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.29.230.60:36097)
0810 15:20:20.844113 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:20.844115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:20.844173 (+ 58us) server_negotiation.cc:407] Connection header received
0810 15:20:20.844230 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:20.844231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:20.844240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:20.844296 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:20.844996 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.845710 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.846626 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:20.846850 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:20.846897 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:20.847030 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:20.847064 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:20.847107 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:20.847297 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 15:20:20.847321 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 15:20:20.848536 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5147
I20260810 15:20:20.864414 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5273
I20260810 15:20:20.873190 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5403
2026-08-10T15:20:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2962 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 15:20:20 dist-test-slave-gjrz krb5kdc[5783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 15:20:20 dist-test-slave-gjrz krb5kdc[5783](info): set up 2 sockets
Aug 10 15:20:20 dist-test-slave-gjrz krb5kdc[5783](info): commencing operation
krb5kdc: starting...
W20260810 15:20:22.052664 30616 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/test-user.keytab.
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T15:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T15:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.230.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
Entry for principal kudu/127.29.230.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab.
I20260810 15:20:22.174104 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42243
--webserver_interface=127.29.230.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 15:20:22.261078 5814 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:22.261296 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:22.261322 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:22.261349 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:22.262486 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:22.262527 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:22.262547 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:22.262562 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:22.262576 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:22.264148 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.62.keytab
--principal=kudu/127.29.230.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.62:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.230.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:22.264392 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:22.264597 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:22.269325 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.269351 5814 server_base.cc:1034] running on GCE node
W20260810 15:20:22.269312 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:22.269299 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.269850 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:22.270141 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:22.271301 5814 hybrid_clock.cc:648] HybridClock initialized: now 1786375222271280 us; error 28 us; skew 500 ppm
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:22.276793 5814 init.cc:377] Logged in from keytab as kudu/127.29.230.62@KRBTEST.COM (short username kudu)
I20260810 15:20:22.277222 5814 webserver.cc:460] Webserver started at http://127.29.230.62:36985/ using document root <none> and password file <none>
I20260810 15:20:22.277420 5814 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:22.277493 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:22.277617 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:22.278702 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data/instance:
uuid: "c1a3bb90b4294501a751073abe2f26ff"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "3fa211d88461293a7ec485753222cf2b"
server_key_iv: "a081f499ffa8d59677c74180ea4f8649"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.279039 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance:
uuid: "c1a3bb90b4294501a751073abe2f26ff"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "3fa211d88461293a7ec485753222cf2b"
server_key_iv: "a081f499ffa8d59677c74180ea4f8649"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.280439 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 15:20:22.281206 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:22.281388 5814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:22.281482 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal
uuid: "c1a3bb90b4294501a751073abe2f26ff"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "3fa211d88461293a7ec485753222cf2b"
server_key_iv: "a081f499ffa8d59677c74180ea4f8649"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.281558 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:22.292788 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:22.294936 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:22.295078 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:22.298974 5814 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.62:42243
I20260810 15:20:22.299021 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.62:42243 every 8 connection(s)
I20260810 15:20:22.299378 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/data/info.pb
I20260810 15:20:22.300360 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:22.300598 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5814
I20260810 15:20:22.300683 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-0/wal/instance
I20260810 15:20:22.300853 30616 external_mini_cluster.cc:1428] Setting key 15883bf2ae4b031054eeaf5f1808e501
I20260810 15:20:22.301985 5940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.61@KRBTEST.COM; defaulting to no policy
I20260810 15:20:22.311826 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.302647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42243 (local address 127.0.0.1:54370)
0810 15:20:22.302811 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.302849 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.303339 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.303630 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.303633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.303866 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.304025 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.304029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.305449 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.305450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.306143 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.306145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.306806 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.306807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.306868 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.307355 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.307364 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.308711 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.310413 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.310419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.310421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.310616 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.310827 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.310827 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.310828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.310896 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.311365 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.311370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.311429 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.311502 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.311643 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260810 15:20:22.311829 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.302691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.29.230.62:42243)
0810 15:20:22.302861 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.302866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.302929 (+ 63us) server_negotiation.cc:407] Connection header received
0810 15:20:22.303505 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.303507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.303529 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.303632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.304118 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.305359 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.306227 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.306714 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.306834 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.308796 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.308805 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.308805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.308813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.310347 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.310687 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.310690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.310690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.310770 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.310947 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.310948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.310948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.311231 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.311388 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.311542 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.311726 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.61@KRBTEST.COM" created.
I20260810 15:20:22.312618 5940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:22.312958 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.312858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35206)
0810 15:20:22.312900 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.312941 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 15:20:22.313077 5842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
I20260810 15:20:22.313474 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.313388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35212)
0810 15:20:22.313429 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.313452 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:22.313573 5940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:44585: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
Entry for principal kudu/127.29.230.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab.
I20260810 15:20:22.327842 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:44585
--webserver_interface=127.29.230.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:22.362797 5940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } attempt: 1
I20260810 15:20:22.363351 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.363182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35224)
0810 15:20:22.363241 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.363316 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 15:20:22.363727 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.363634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35236)
0810 15:20:22.363677 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.363705 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:22.363839 5940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:44585: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
W20260810 15:20:22.409636 5949 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:22.409850 5949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:22.409874 5949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:22.409900 5949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:22.411003 5949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:22.411043 5949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:22.411062 5949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:22.411078 5949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:22.411095 5949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:22.412635 5949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.61.keytab
--principal=kudu/127.29.230.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.61:44585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.230.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:22.412889 5949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:22.413103 5949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:22.417686 5967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:22.417733 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.417901 5949 server_base.cc:1034] running on GCE node
W20260810 15:20:22.417701 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.418171 5949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:22.418437 5949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:22.419569 5949 hybrid_clock.cc:648] HybridClock initialized: now 1786375222419551 us; error 33 us; skew 500 ppm
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:22.421926 5949 init.cc:377] Logged in from keytab as kudu/127.29.230.61@KRBTEST.COM (short username kudu)
I20260810 15:20:22.422334 5949 webserver.cc:460] Webserver started at http://127.29.230.61:38089/ using document root <none> and password file <none>
I20260810 15:20:22.422530 5949 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:22.422569 5949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:22.422655 5949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:22.423713 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data/instance:
uuid: "ed1c63f951564c7c8b05d39de59546b0"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "67825b0452beb298dd2060050f7a1ef1"
server_key_iv: "e1882b063757570bbb91a38de9b7d9fb"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.424009 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal/instance:
uuid: "ed1c63f951564c7c8b05d39de59546b0"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "67825b0452beb298dd2060050f7a1ef1"
server_key_iv: "e1882b063757570bbb91a38de9b7d9fb"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.425184 5949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 15:20:22.425829 5979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:22.425988 5949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:22.426069 5949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal
uuid: "ed1c63f951564c7c8b05d39de59546b0"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "67825b0452beb298dd2060050f7a1ef1"
server_key_iv: "e1882b063757570bbb91a38de9b7d9fb"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.426124 5949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:22.431991 5940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } attempt: 2
I20260810 15:20:22.432404 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.432284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35238)
0810 15:20:22.432337 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.432375 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 15:20:22.432677 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.432609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35252)
0810 15:20:22.432639 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.432661 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260810 15:20:22.432757 5940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.61:44585: Network error: Client connection negotiation failed: client connection to 127.29.230.61:44585: connect: Connection refused (error 111)
I20260810 15:20:22.438154 5949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:22.440083 5949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:22.440209 5949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:22.443924 5949 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.61:44585
I20260810 15:20:22.443986 6073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.61:44585 every 8 connection(s)
I20260810 15:20:22.444284 5949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/data/info.pb
I20260810 15:20:22.445262 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:22.446749 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:22.453269 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 5949
I20260810 15:20:22.453356 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-1/wal/instance
I20260810 15:20:22.453534 30616 external_mini_cluster.cc:1428] Setting key 4da8712e789498b2f70a4a2f255034db
I20260810 15:20:22.455675 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.447453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.29.230.62:42243)
0810 15:20:22.447512 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.447514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.447695 (+ 181us) server_negotiation.cc:407] Connection header received
0810 15:20:22.448183 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.448184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.448200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.448252 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.448843 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.449336 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.450302 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.450507 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.450565 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.452786 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.452787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.452788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.452797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.454354 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.454747 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.454748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.454748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.454821 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.455019 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.455019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.455020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.455160 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.455224 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.455498 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.455588 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:20:22.455855 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.447431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42243 (local address 127.0.0.1:54376)
0810 15:20:22.447580 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.447615 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.448080 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.448319 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.448322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.448565 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.448749 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.448754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.449426 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.449427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.450209 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.450212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.450589 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.450591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.450726 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.451216 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.451224 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.452685 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.454435 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.454443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.454445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.454677 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.454878 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.454879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.454879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.454969 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.455215 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.455218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.455369 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.455461 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.455634 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260810 15:20:22.456373 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.230.60@KRBTEST.COM; defaulting to no policy
I20260810 15:20:22.463992 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.456656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35264)
0810 15:20:22.456697 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.456707 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.456745 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.457100 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.457102 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.457233 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.457295 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.457297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.458187 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.458189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.458939 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.458942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.459609 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.459610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.459675 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.460073 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.460076 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.461389 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.463151 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.463152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.463153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.463329 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.463490 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.463490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.463490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.463535 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.463765 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.463766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.463801 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.463851 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.463927 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 15:20:22.464111 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.456790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.29.230.61:44585)
0810 15:20:22.456950 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.456954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.456973 (+ 19us) server_negotiation.cc:407] Connection header received
0810 15:20:22.456992 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.457001 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.457014 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.457080 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.457384 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.458088 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.459018 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.459527 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.459606 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.461472 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.461482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.461482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.461489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.463070 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.463393 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.463396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.463396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.463448 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.463572 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.463572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.463572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.463714 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.463773 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.463874 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.464005 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260810 15:20:22.464717 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:22.465044 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.464947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34536)
0810 15:20:22.464986 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.465025 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 15:20:22.465224 5982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.230.60@KRBTEST.COM" created.
I20260810 15:20:22.465641 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.465564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34540)
0810 15:20:22.465600 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.465624 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260810 15:20:22.465754 6074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:37237: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
Entry for principal kudu/127.29.230.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab.
I20260810 15:20:22.480250 30616 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX3gT9X/build/release/bin/kudu
/tmp/dist-test-taskX3gT9X/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:37237
--webserver_interface=127.29.230.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.230.20:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 15:20:22.514995 6074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } attempt: 1
I20260810 15:20:22.515554 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.515429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34548)
0810 15:20:22.515488 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.515521 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 15:20:22.515913 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.515820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34560)
0810 15:20:22.515862 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.515893 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:22.516002 6074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:37237: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
I20260810 15:20:22.523895 5940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } attempt: 3
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:22.530596 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.524271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.29.230.61:44585)
0810 15:20:22.524329 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.524331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.524357 (+ 26us) server_negotiation.cc:407] Connection header received
0810 15:20:22.524382 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.524383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.524394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.524435 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.524768 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.525256 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.525938 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.526157 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.526219 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.528044 (+ 1825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.528046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.528046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.528054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.529384 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.529822 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.529823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.529823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.529929 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.530148 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.530149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.530149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.530270 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.530329 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.530472 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.530532 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 15:20:22.530658 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.524207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35266)
0810 15:20:22.524265 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.524275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.524321 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.524471 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.524472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.524625 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.524690 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.524692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.525329 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.525330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.525856 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.525859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.526220 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.526221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.526300 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.526763 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.526767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.527966 (+ 1199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.529511 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.529512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.529513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.529732 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.529995 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.529996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.529996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.530067 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.530329 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.530330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.530385 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.530453 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.530562 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:20:22.531086 5940 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 15:20:22.531421 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.531324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34572)
0810 15:20:22.531360 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.531397 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 15:20:22.531747 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.531663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34586)
0810 15:20:22.531704 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.531730 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 15:20:22.531831 5940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:37237: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
I20260810 15:20:22.562981 5940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } attempt: 1
I20260810 15:20:22.563504 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.563358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34600)
0810 15:20:22.563428 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.563472 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 15:20:22.563843 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.563760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34608)
0810 15:20:22.563799 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.563824 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 15:20:22.563944 5940 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:37237: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
W20260810 15:20:22.563959 6085 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 15:20:22.564172 6085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 15:20:22.564193 6085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 15:20:22.564213 6085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 15:20:22.565397 6085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 15:20:22.565438 6085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 15:20:22.565451 6085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 15:20:22.565462 6085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 15:20:22.565474 6085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 15:20:22.566955 6085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.230.20:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.230.62:42243,127.29.230.61:44585,127.29.230.60:37237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/krb5kdc/kudu_127.29.230.60.keytab
--principal=kudu/127.29.230.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.230.60:37237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.230.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 14:17:30 UTC on ccycloud.quasar-yaaqgv.root.comops.site
build id 4577963
I20260810 15:20:22.567162 6085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 15:20:22.567384 6085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 15:20:22.571997 6100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 15:20:22.571978 6097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.572237 6085 server_base.cc:1034] running on GCE node
W20260810 15:20:22.572024 6096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 15:20:22.573271 6085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 15:20:22.573576 6085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 15:20:22.574746 6085 hybrid_clock.cc:648] HybridClock initialized: now 1786375222574727 us; error 30 us; skew 500 ppm
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 15:20:22.577291 6085 init.cc:377] Logged in from keytab as kudu/127.29.230.60@KRBTEST.COM (short username kudu)
I20260810 15:20:22.577716 6085 webserver.cc:460] Webserver started at http://127.29.230.60:45757/ using document root <none> and password file <none>
I20260810 15:20:22.577939 6085 fs_manager.cc:359] Metadata directory not provided
I20260810 15:20:22.577996 6085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 15:20:22.578123 6085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 15:20:22.579152 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data/instance:
uuid: "bf403b15652b404d80bd00d2eac89058"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "8aa09fa37eaa83535754184f35e1f0fa"
server_key_iv: "ad6fd063726f2c8b0ce0fd65217fe95f"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.579555 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal/instance:
uuid: "bf403b15652b404d80bd00d2eac89058"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "8aa09fa37eaa83535754184f35e1f0fa"
server_key_iv: "ad6fd063726f2c8b0ce0fd65217fe95f"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.580793 6085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 15:20:22.581547 6116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 15:20:22.581727 6085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 15:20:22.581825 6085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data,/tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal
uuid: "bf403b15652b404d80bd00d2eac89058"
format_stamp: "Formatted at 2026-08-10 15:20:22 on dist-test-slave-gjrz"
server_key: "8aa09fa37eaa83535754184f35e1f0fa"
server_key_iv: "ad6fd063726f2c8b0ce0fd65217fe95f"
server_key_version: "encryptionkey@0"
I20260810 15:20:22.581900 6085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 15:20:22.584129 6074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } attempt: 2
I20260810 15:20:22.584481 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.584368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34624)
0810 15:20:22.584418 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.584449 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 15:20:22.584843 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.584753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34638)
0810 15:20:22.584796 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.584820 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 15:20:22.584924 6074 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.230.60:37237: Network error: Client connection negotiation failed: client connection to 127.29.230.60:37237: connect: Connection refused (error 111)
I20260810 15:20:22.600271 6085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 15:20:22.602162 6085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 15:20:22.602295 6085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 15:20:22.606068 6085 rpc_server.cc:307] RPC server started. Bound to: 127.29.230.60:37237
I20260810 15:20:22.606115 6202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.230.60:37237 every 8 connection(s)
I20260810 15:20:22.606427 6085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/data/info.pb
I20260810 15:20:22.607424 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 15:20:22.608945 6203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:22.615859 30616 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX3gT9X/build/release/bin/kudu as pid 6085
I20260810 15:20:22.615952 30616 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX3gT9X/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786374947162751-30616-0/minicluster-data/master-2/wal/instance
I20260810 15:20:22.616140 30616 external_mini_cluster.cc:1428] Setting key a08ab5895480a9797d7e32651fcbdad0
I20260810 15:20:22.617828 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.609745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.29.230.62:42243)
0810 15:20:22.609818 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.609820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.609985 (+ 165us) server_negotiation.cc:407] Connection header received
0810 15:20:22.610463 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.610464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.610475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.610527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.611125 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.611667 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.612665 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.612856 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.612913 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.614990 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.614992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.614992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.615000 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.616337 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.616782 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.616783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.616783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.616875 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.617160 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.617160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.617161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.617272 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.617318 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.617649 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.617735 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 15:20:22.617978 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.609690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.62:42243 (local address 127.0.0.1:54392)
0810 15:20:22.609855 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.609897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.610356 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.610589 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.610592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.610827 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.611020 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.611024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.611767 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.611768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.612583 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.612586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.612927 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.612928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.613048 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.613520 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.613529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.614907 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.616408 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.616415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.616418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.616714 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.616937 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.616938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.616938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.617092 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.617327 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.617333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.617499 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.617619 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.617747 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260810 15:20:22.618469 6203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:22.624784 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.618772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.29.230.61:44585)
0810 15:20:22.618839 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.618841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.618848 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:22.618871 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.618872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.618882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.618925 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.619176 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.619874 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.620500 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.620716 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.620769 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.622437 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.622438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.622439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.622446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.623743 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.624034 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.624035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.624035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.624095 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.624284 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.624285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.624285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.624411 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.624474 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.624612 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.624693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 15:20:22.624778 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.618702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.61:44585 (local address 127.0.0.1:35270)
0810 15:20:22.618746 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.618756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.618785 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.618934 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.618934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.619067 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.619109 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.619110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.619955 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.619956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.620426 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.620427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.620775 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.620776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.620842 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.621203 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.621208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.622374 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.623802 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.623802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.623803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.623974 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.624161 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.624162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.624162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.624226 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.624475 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.624476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.624519 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.624584 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.624672 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 15:20:22.624864 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.616549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34646 (local address 127.29.230.60:37237)
0810 15:20:22.616691 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.616696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.616740 (+ 44us) server_negotiation.cc:407] Connection header received
0810 15:20:22.616811 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.616815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.616829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.616901 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.617570 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.618525 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.619332 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.619780 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.619847 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.622158 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.622168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.622168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.622175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.623669 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.624047 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.624050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.624051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.624106 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.624284 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.624285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.624285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.624447 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.624578 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.624661 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.624731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260810 15:20:22.625174 6203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.60@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.62@KRBTEST.COM
I20260810 15:20:22.631311 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.625411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34648)
0810 15:20:22.625452 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.625464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.625498 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.625634 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.625635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.625733 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.625761 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.625762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.626342 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.626343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.626797 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.626799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.627116 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.627117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.627162 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.627504 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.627509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.628756 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.630302 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.630303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.630304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.630492 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.630683 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.630684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.630685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.630745 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.630947 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.630948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.630991 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.631069 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.631163 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":1}
I20260810 15:20:22.631325 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.625456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.29.230.60:37237)
0810 15:20:22.625491 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.625492 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.625500 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:22.625558 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.625559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.625570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.625633 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.625823 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.626280 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.626859 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.627061 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.627113 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.628835 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.628837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.628837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.628844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.630222 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.630562 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.630563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.630563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.630622 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.630803 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.630803 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.630804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.630895 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.630937 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.631085 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.631202 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":19}
I20260810 15:20:22.632347 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.625771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.29.230.62:42243)
0810 15:20:22.625808 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.625809 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.625816 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:22.625836 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.625837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.625845 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.625890 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.626318 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.626765 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.627452 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.627643 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.627693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.629798 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.629799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.629799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.629806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.631212 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.631605 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.631606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.631607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.631665 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.631856 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.631857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.631857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.632007 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.632127 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.632225 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.632288 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:22.633050 6203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: Bootstrap starting.
I20260810 15:20:22.633994 6203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:22.634294 6203 log.cc:824] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:22.635046 6203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: No bootstrap required, opened a new log
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.230.61@KRBTEST.COM
I20260810 15:20:22.636940 6203 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.637090 6203 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf403b15652b404d80bd00d2eac89058, State: Initialized, Role: FOLLOWER
I20260810 15:20:22.637177 6203 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.637480 6203 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:22.637661 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.637750 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.639176 5940 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } attempt: 2
W20260810 15:20:22.639379 6232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:22.639496 6232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:22.640343 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.632885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.29.230.61:44585)
0810 15:20:22.632920 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.632922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.632930 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:22.632997 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.632998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.633006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.633044 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.633571 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.634019 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.634844 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.635037 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.635085 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.637361 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.637362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.637363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.637370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.638914 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.639321 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.639322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.639323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.639382 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.639570 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.639571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.639572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.639695 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.639834 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.640188 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.640260 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 15:20:22.639936 6085 master_runner.cc:186] Error getting master registration for 127.29.230.62:42243: OK, instance_id { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" instance_seqno: 1786375222297930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:20:22.642817 6085 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.62@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:22.644104 30616 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 15:20:22.645226 5883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c1a3bb90b4294501a751073abe2f26ff"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54402
I20260810 15:20:22.646968 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.639829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.29.230.60:37237)
0810 15:20:22.639864 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.639866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.639873 (+ 7us) server_negotiation.cc:407] Connection header received
0810 15:20:22.639889 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.639890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.639898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.639941 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.640324 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.640989 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.641961 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.642228 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.642298 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.644246 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.644248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.644248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.644254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.645788 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.646187 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.646189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.646189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.646252 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.646442 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.646443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.646443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.646601 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.646663 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.646807 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.646882 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 15:20:22.646968 5944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.639500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34654)
0810 15:20:22.639646 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.639655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.639692 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.639982 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.639983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.640138 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.640181 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.640183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.641089 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.641091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.641859 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.641862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.642306 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.642307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.642370 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.642699 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.642704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.644152 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.645881 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.645882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.645882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.646106 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.646321 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.646321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.646322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.646384 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.646662 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.646663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.646710 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.646781 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.646873 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 15:20:22.648370 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: Bootstrap starting.
I20260810 15:20:22.649091 5940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:22.649399 5940 log.cc:824] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:22.650252 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: No bootstrap required, opened a new log
I20260810 15:20:22.651577 5940 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.651714 5940 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1a3bb90b4294501a751073abe2f26ff, State: Initialized, Role: FOLLOWER
I20260810 15:20:22.651795 5940 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.652118 5940 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:22.652151 5883 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 15:20:22.652202 5883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:22.653014 5883 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.653270 5883 leader_election.cc:288] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [CANDIDATE]: Term 1 election: Requested vote from peers ed1c63f951564c7c8b05d39de59546b0 (127.29.230.61:44585), bf403b15652b404d80bd00d2eac89058 (127.29.230.60:37237)
I20260810 15:20:22.653820 6161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1a3bb90b4294501a751073abe2f26ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bf403b15652b404d80bd00d2eac89058"
I20260810 15:20:22.653942 6161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [term 0 FOLLOWER]: Advancing to term 1
I20260810 15:20:22.654774 6161 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a3bb90b4294501a751073abe2f26ff in term 1.
I20260810 15:20:22.652081 6237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.655510 6237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: This master's current role is: FOLLOWER
W20260810 15:20:22.656616 6268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:22.656685 6268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:22.654911 5814 master_runner.cc:186] Error getting master registration for 127.29.230.61:44585: OK, instance_id { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" instance_seqno: 1786375222442857 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 15:20:22.657564 5814 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 15:20:22.657776 5842 leader_election.cc:302] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf403b15652b404d80bd00d2eac89058, c1a3bb90b4294501a751073abe2f26ff; no voters:
I20260810 15:20:22.657910 6237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 15:20:22.658170 6237 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [term 1 LEADER]: Becoming Leader. State: Replica: c1a3bb90b4294501a751073abe2f26ff, State: Running, Role: LEADER
I20260810 15:20:22.658310 6237 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.658960 6237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1a3bb90b4294501a751073abe2f26ff. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.659144 6237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:22.664000 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.655799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.29.230.62:42243)
0810 15:20:22.655836 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.655838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.656142 (+ 304us) server_negotiation.cc:407] Connection header received
0810 15:20:22.656244 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.656246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.656256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.656285 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.657063 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.657740 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.658765 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.659025 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.659085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.660547 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.660549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.660549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.660556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.662281 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.662639 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.662640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.662640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.662940 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.663179 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.663180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.663180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.663333 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.663442 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.663846 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.663916 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:22.667704 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.658175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.29.230.61:44585)
0810 15:20:22.658216 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.658218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.658800 (+ 582us) server_negotiation.cc:407] Connection header received
0810 15:20:22.658893 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.658894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.658904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.658932 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.659793 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.660512 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.661511 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.662379 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.662424 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.663977 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.663978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.663978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.663985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.665504 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.665861 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.665862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.665863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.665930 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.666945 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.666946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.666946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.667090 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.667178 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.667520 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.667613 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:22.671875 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.658582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34670 (local address 127.29.230.60:37237)
0810 15:20:22.658720 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.658721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.662802 (+ 4081us) server_negotiation.cc:407] Connection header received
0810 15:20:22.662912 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.662913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.662923 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.662951 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.664553 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.665505 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.666969 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.667259 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.667319 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.669158 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.669159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.669160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.669167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.670690 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.671065 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.671066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.671066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.671124 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.671350 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.671351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.671351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.671480 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.671585 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.671709 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.671784 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 15:20:22.675804 6289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 15:20:22.675998 6289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 15:20:22.676064 6074 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } attempt: 3
Aug 10 15:20:22 dist-test-slave-gjrz krb5kdc[5783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786375222, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.230.61@KRBTEST.COM for kudu/127.29.230.60@KRBTEST.COM
I20260810 15:20:22.683503 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.676302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.230.60:37237 (local address 127.0.0.1:34680)
0810 15:20:22.676344 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 15:20:22.676355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 15:20:22.676403 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 15:20:22.676537 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 15:20:22.676538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 15:20:22.676693 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 15:20:22.676737 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.676739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.677611 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.677612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.678414 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 15:20:22.678416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.678884 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.678885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 15:20:22.678939 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.679394 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 15:20:22.679398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 15:20:22.680808 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 15:20:22.682468 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.682469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.682469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.682664 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.682849 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 15:20:22.682850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 15:20:22.682850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 15:20:22.682902 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 15:20:22.683174 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 15:20:22.683175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 15:20:22.683220 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 15:20:22.683309 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 15:20:22.683404 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260810 15:20:22.684001 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:22.676342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.29.230.60:37237)
0810 15:20:22.676379 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.676380 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.676388 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:22.676447 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.676448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.676456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.676499 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.676811 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.677522 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.678501 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:22.678785 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:22.678839 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:22.680905 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 15:20:22.680906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 15:20:22.680907 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 15:20:22.680913 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 15:20:22.682379 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.682735 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.682736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.682736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.682788 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 15:20:22.682962 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 15:20:22.682963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 15:20:22.682963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 15:20:22.683091 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 15:20:22.683148 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:22.683798 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 15:20:22.683872 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:22.684458 6161 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [term 1 FOLLOWER]: Refusing update from remote peer c1a3bb90b4294501a751073abe2f26ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:22.684806 6281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:22.685654 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: Bootstrap starting.
I20260810 15:20:22.686379 6074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: Neither blocks nor log segments found. Creating new log.
I20260810 15:20:22.686655 6074 log.cc:824] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: Log is configured to *not* fsync() on all Append() calls
I20260810 15:20:22.686892 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1a3bb90b4294501a751073abe2f26ff. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.686966 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.687618 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: No bootstrap required, opened a new log
I20260810 15:20:22.689524 6074 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.689653 6074 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1c63f951564c7c8b05d39de59546b0, State: Initialized, Role: FOLLOWER
I20260810 15:20:22.689723 6074 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } }
I20260810 15:20:22.690044 6074 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 15:20:22.690477 6023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1a3bb90b4294501a751073abe2f26ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ed1c63f951564c7c8b05d39de59546b0"
I20260810 15:20:22.690585 6023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [term 0 FOLLOWER]: Advancing to term 1
W20260810 15:20:22.690762 5989 tablet.cc:2369] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 15:20:22.691462 6023 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1a3bb90b4294501a751073abe2f26ff in term 1.
I20260810 15:20:22.691679 6021 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [term 1 FOLLOWER]: Refusing update from remote peer c1a3bb90b4294501a751073abe2f26ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 15:20:22.691984 6281 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 15:20:22.693712 6281 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.693817 6281 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:22.693900 6290 mvcc.cc:205] Tried to move back new op lower bound from 7316992912113090560 to 7316992912009678848. Current Snapshot: MvccSnapshot[applied={T|T < 7316992912113090560}]
I20260810 15:20:22.694115 6237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.694192 6237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff [sys.catalog]: This master's current role is: LEADER
I20260810 15:20:22.694628 6289 catalog_manager.cc:1347] Generated new cluster ID: 5ffcdc7145ea4c5088d7518bb5e4ad69
I20260810 15:20:22.694674 6289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 15:20:22.694687 6211 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.694764 6211 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.695017 6292 mvcc.cc:205] Tried to move back new op lower bound from 7316992912113090560 to 7316992912009678848. Current Snapshot: MvccSnapshot[applied={T|T < 7316992912113090560}]
W20260810 15:20:22.699782 6317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 15:20:22.699841 6317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 15:20:22.703383 6320 mvcc.cc:205] Tried to move back new op lower bound from 7316992912113090560 to 7316992912009678848. Current Snapshot: MvccSnapshot[applied={T|T < 7316992912113090560}]
I20260810 15:20:22.703356 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.703509 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.703578 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1a3bb90b4294501a751073abe2f26ff. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.703629 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.703658 6295 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1a3bb90b4294501a751073abe2f26ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1a3bb90b4294501a751073abe2f26ff" member_type: VOTER last_known_addr { host: "127.29.230.62" port: 42243 } } peers { permanent_uuid: "ed1c63f951564c7c8b05d39de59546b0" member_type: VOTER last_known_addr { host: "127.29.230.61" port: 44585 } } peers { permanent_uuid: "bf403b15652b404d80bd00d2eac89058" member_type: VOTER last_known_addr { host: "127.29.230.60" port: 37237 } } }
I20260810 15:20:22.703697 6295 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 15:20:22.727337 6289 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 15:20:22.729226 6289 catalog_manager.cc:1504] Loading token signing keys...
I20260810 15:20:22.740366 6289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1a3bb90b4294501a751073abe2f26ff: Generated new TSK 0
I20260810 15:20:22.740675 6289 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 15:20:22.772009 5946 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.770247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.29.230.62:42243)
0810 15:20:22.770295 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.770297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.771312 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 15:20:22.771420 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.771422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.771434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.771465 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.771935 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54416: BlockingRecv error: recv got EOF from 127.0.0.1:54416 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 15:20:22.773000 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.770698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.29.230.61:44585)
0810 15:20:22.770738 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.770739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.772469 (+ 1730us) server_negotiation.cc:407] Connection header received
0810 15:20:22.772565 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.772566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.772576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.772606 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.772934 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35300: BlockingRecv error: recv got EOF from 127.0.0.1:35300 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:20:22.773876 6208 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.771002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.29.230.60:37237)
0810 15:20:22.771052 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.771054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.773350 (+ 2296us) server_negotiation.cc:407] Connection header received
0810 15:20:22.773472 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.773473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.773484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.773512 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.773818 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34682: BlockingRecv error: recv got EOF from 127.0.0.1:34682 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 15:20:22.776427 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.775630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.29.230.61:44585)
0810 15:20:22.775666 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.775668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.775877 (+ 209us) server_negotiation.cc:407] Connection header received
0810 15:20:22.775981 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.775982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.776000 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.776041 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.776380 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35306: BlockingRecv error: recv got EOF from 127.0.0.1:35306 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:20:22.779541 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.778761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35318 (local address 127.29.230.61:44585)
0810 15:20:22.778794 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.778795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.779025 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:20:22.779126 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.779127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.779137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.779164 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.779495 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35318: BlockingRecv error: recv got EOF from 127.0.0.1:35318 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 15:20:22.783638 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.782740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.29.230.61:44585)
0810 15:20:22.782772 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.782773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.783007 (+ 234us) server_negotiation.cc:407] Connection header received
0810 15:20:22.783115 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.783116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.783126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.783155 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.783593 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35322: BlockingRecv error: recv got EOF from 127.0.0.1:35322 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 15:20:22.789574 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.788863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.29.230.61:44585)
0810 15:20:22.788897 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.788898 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.789105 (+ 207us) server_negotiation.cc:407] Connection header received
0810 15:20:22.789189 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.789190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.789200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.789226 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.789526 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35326: BlockingRecv error: recv got EOF from 127.0.0.1:35326 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260810 15:20:22.799773 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.798897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.29.230.61:44585)
0810 15:20:22.798933 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.798935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.799165 (+ 230us) server_negotiation.cc:407] Connection header received
0810 15:20:22.799253 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.799254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.799351 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.799381 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.799721 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 15:20:22.817835 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.817184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.29.230.61:44585)
0810 15:20:22.817227 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.817229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.817338 (+ 109us) server_negotiation.cc:407] Connection header received
0810 15:20:22.817396 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.817397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.817409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.817457 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.817774 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35352: BlockingRecv error: recv got EOF from 127.0.0.1:35352 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 15:20:22.851665 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.850912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.29.230.61:44585)
0810 15:20:22.850962 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.850964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.851072 (+ 108us) server_negotiation.cc:407] Connection header received
0810 15:20:22.851146 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.851148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.851160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.851202 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.851602 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35362: BlockingRecv error: recv got EOF from 127.0.0.1:35362 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260810 15:20:22.917510 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:22.916847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.29.230.61:44585)
0810 15:20:22.916940 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:22.916942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:22.917017 (+ 75us) server_negotiation.cc:407] Connection header received
0810 15:20:22.917053 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:22.917054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:22.917065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:22.917123 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:22.917440 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35370: BlockingRecv error: recv got EOF from 127.0.0.1:35370 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 15:20:23.047444 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:23.046695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.29.230.61:44585)
0810 15:20:23.046753 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.046756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.046857 (+ 101us) server_negotiation.cc:407] Connection header received
0810 15:20:23.046937 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.046938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.046971 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.047016 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:23.047377 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35378: BlockingRecv error: recv got EOF from 127.0.0.1:35378 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 15:20:23.305326 6081 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 15:20:23.304663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.29.230.61:44585)
0810 15:20:23.304726 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.304728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.304739 (+ 11us) server_negotiation.cc:407] Connection header received
0810 15:20:23.304832 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.304833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.304844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.304908 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 15:20:23.305280 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35388: BlockingRecv error: recv got EOF from 127.0.0.1:35388 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 15:20:23.641708 6232 catalog_manager.cc:1259] Loaded cluster ID: 5ffcdc7145ea4c5088d7518bb5e4ad69
I20260810 15:20:23.641783 6232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: loading cluster ID for follower catalog manager: success
I20260810 15:20:23.642612 6232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: acquiring CA information for follower catalog manager: success
I20260810 15:20:23.642931 6232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bf403b15652b404d80bd00d2eac89058: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:20:23.700284 6317 catalog_manager.cc:1259] Loaded cluster ID: 5ffcdc7145ea4c5088d7518bb5e4ad69
I20260810 15:20:23.700364 6317 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: loading cluster ID for follower catalog manager: success
I20260810 15:20:23.701190 6317 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: acquiring CA information for follower catalog manager: success
I20260810 15:20:23.701500 6317 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ed1c63f951564c7c8b05d39de59546b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 15:20:23.821322 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:23.818537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.29.230.61:44585)
0810 15:20:23.818640 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.818642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.818672 (+ 30us) server_negotiation.cc:407] Connection header received
0810 15:20:23.818699 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.818700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.818751 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.818814 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:23.819045 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.819746 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.820568 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.820774 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.820832 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:23.821012 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:23.821133 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:23.821189 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:23.821209 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 15:20:23.821229 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260810 15:20:23.825393 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:23.823055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.29.230.60:37237)
0810 15:20:23.823179 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.823181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.823189 (+ 8us) server_negotiation.cc:407] Connection header received
0810 15:20:23.823231 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.823232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.823310 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.823380 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:23.823492 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.823983 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.824682 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.824883 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.824935 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:23.825077 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:23.825181 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:23.825245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:23.825291 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 15:20:23.825312 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260810 15:20:23.829679 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:23.826926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.29.230.61:44585)
0810 15:20:23.826964 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.826966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.827101 (+ 135us) server_negotiation.cc:407] Connection header received
0810 15:20:23.827189 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.827191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.827201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.827251 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:23.827433 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.828194 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.828982 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.829282 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.829350 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:23.829468 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:23.829500 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:23.829546 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:23.829588 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 15:20:23.829608 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 15:20:23.829782 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:23.827215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.29.230.60:37237)
0810 15:20:23.827251 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.827253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.827337 (+ 84us) server_negotiation.cc:407] Connection header received
0810 15:20:23.827392 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.827393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.827403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.827444 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:23.827578 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.828043 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.829019 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.829297 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.829350 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:23.829519 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:23.829556 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:23.829598 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:23.829694 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 15:20:23.829715 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 15:20:23.830258 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 15:20:23.827012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.29.230.62:42243)
0810 15:20:23.827122 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 15:20:23.827124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 15:20:23.827247 (+ 123us) server_negotiation.cc:407] Connection header received
0810 15:20:23.827311 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 15:20:23.827312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 15:20:23.827360 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 15:20:23.827438 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 15:20:23.827546 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.828345 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.829280 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 15:20:23.829575 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 15:20:23.829629 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 15:20:23.829837 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 15:20:23.829944 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 15:20:23.830000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 15:20:23.830056 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 15:20:23.830200 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260810 15:20:23.831763 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5814
I20260810 15:20:23.842558 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 5949
I20260810 15:20:23.853727 30616 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX3gT9X/build/release/bin/kudu with pid 6085
2026-08-10T15:20:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2982 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5945 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (276693 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST